[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707036041078-13712-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:37:16 dist-test-slave-j1xs krb5kdc[13719](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:37:16 dist-test-slave-j1xs krb5kdc[13719](info): set up 2 sockets
Sep 29 18:37:16 dist-test-slave-j1xs krb5kdc[13719](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260929 18:37:17.098846 13712 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707036041078-13712-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707036041078-13712-0/krb5kdc/test-user.keytab.
Sep 29 18:37:17 dist-test-slave-j1xs krb5kdc[13719](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707037, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:37:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:37:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.100.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.100.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.100.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.62.keytab.
Entry for principal kudu/127.13.100.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.62.keytab.
I20260929 18:37:17.225812 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707036041078-13712-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707036041078-13712-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707036041078-13712-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.100.62:35197
--webserver_interface=127.13.100.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.62.keytab
--principal=kudu/127.13.100.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.100.20:38413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.100.62:35197
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707036041078-13712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707036041078-13712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707036041078-13712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:37:17.331718 13732 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:37:17.332022 13732 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:37:17.332096 13732 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:37:17.336071 13732 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:37:17.336170 13732 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:37:17.336210 13732 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:37:17.336266 13732 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:37:17.336302 13732 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:37:17.341459 13732 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.100.20:38413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707036041078-13712-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707036041078-13712-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.100.62:35197
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.62.keytab
--principal=kudu/127.13.100.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.100.62:35197
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.100.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13732
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707036041078-13712-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:37:17.342571 13732 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:37:17.343617 13732 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:37:17.352301 13737 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:37:17.352274 13740 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:37:17.352403 13732 server_base.cc:1034] running on GCE node
W20260929 18:37:17.352280 13738 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:37:17.353039 13732 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:37:17.354231 13732 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:37:17.355406 13732 hybrid_clock.cc:648] HybridClock initialized: now 1790707037355392 us; error 35 us; skew 500 ppm
Sep 29 18:37:17 dist-test-slave-j1xs krb5kdc[13719](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707037, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:37:17.358244 13732 init.cc:377] Logged in from keytab as kudu/127.13.100.62@KRBTEST.COM (short username kudu)
I20260929 18:37:17.359216 13732 webserver.cc:460] Webserver started at http://127.13.100.62:42583/ using document root <none> and password file <none>
I20260929 18:37:17.359773 13732 fs_manager.cc:359] Metadata directory not provided
I20260929 18:37:17.359833 13732 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:37:17.360064 13732 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:37:17.362977 13732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707036041078-13712-0/minicluster-data/master-0/data/instance:
uuid: "82216e2b806b4f199aeda06b6322afbb"
format_stamp: "Formatted at 2026-09-29 18:37:17 on dist-test-slave-j1xs"
server_key: "10bbc512f8b5c15376a4876e472d1205"
server_key_iv: "5a3bf8ae4a2ba05091c442f0d5ed927f"
server_key_version: "encryptionkey@0"
I20260929 18:37:17.363682 13732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707036041078-13712-0/minicluster-data/master-0/wal/instance:
uuid: "82216e2b806b4f199aeda06b6322afbb"
format_stamp: "Formatted at 2026-09-29 18:37:17 on dist-test-slave-j1xs"
server_key: "10bbc512f8b5c15376a4876e472d1205"
server_key_iv: "5a3bf8ae4a2ba05091c442f0d5ed927f"
server_key_version: "encryptionkey@0"
I20260929 18:37:17.367365 13732 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 18:37:17.369716 13747 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:37:17.371124 13732 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:37:17.371264 13732 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707036041078-13712-0/minicluster-data/master-0/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707036041078-13712-0/minicluster-data/master-0/wal
uuid: "82216e2b806b4f199aeda06b6322afbb"
format_stamp: "Formatted at 2026-09-29 18:37:17 on dist-test-slave-j1xs"
server_key: "10bbc512f8b5c15376a4876e472d1205"
server_key_iv: "5a3bf8ae4a2ba05091c442f0d5ed927f"
server_key_version: "encryptionkey@0"
I20260929 18:37:17.371372 13732 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707036041078-13712-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707036041078-13712-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707036041078-13712-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:37:17.395005 13732 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:37:17.398739 13732 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:37:17.399019 13732 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:37:17.406514 13732 rpc_server.cc:307] RPC server started. Bound to: 127.13.100.62:35197
I20260929 18:37:17.406530 13799 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.100.62:35197 every 8 connection(s)
I20260929 18:37:17.407469 13732 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
I20260929 18:37:17.410266 13800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:37:17.411789 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 13732
I20260929 18:37:17.411969 13712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707036041078-13712-0/minicluster-data/master-0/wal/instance
I20260929 18:37:17.413219 13712 external_mini_cluster.cc:1428] Setting key 3a91ef38d29feb795c8ead446d07382f
I20260929 18:37:17.416165 13800 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82216e2b806b4f199aeda06b6322afbb: Bootstrap starting.
I20260929 18:37:17.418699 13800 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 82216e2b806b4f199aeda06b6322afbb: Neither blocks nor log segments found. Creating new log.
I20260929 18:37:17.419392 13800 log.cc:824] T 00000000000000000000000000000000 P 82216e2b806b4f199aeda06b6322afbb: Log is configured to *not* fsync() on all Append() calls
I20260929 18:37:17.421298 13800 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82216e2b806b4f199aeda06b6322afbb: No bootstrap required, opened a new log
I20260929 18:37:17.424655 13800 raft_consensus.cc:348] T 00000000000000000000000000000000 P 82216e2b806b4f199aeda06b6322afbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82216e2b806b4f199aeda06b6322afbb" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 35197 } }
I20260929 18:37:17.424882 13800 raft_consensus.cc:374] T 00000000000000000000000000000000 P 82216e2b806b4f199aeda06b6322afbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:37:17.424942 13800 raft_consensus.cc:729] T 00000000000000000000000000000000 P 82216e2b806b4f199aeda06b6322afbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82216e2b806b4f199aeda06b6322afbb, State: Initialized, Role: FOLLOWER
I20260929 18:37:17.425396 13800 consensus_queue.cc:260] T 00000000000000000000000000000000 P 82216e2b806b4f199aeda06b6322afbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82216e2b806b4f199aeda06b6322afbb" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 35197 } }
I20260929 18:37:17.425513 13800 raft_consensus.cc:388] T 00000000000000000000000000000000 P 82216e2b806b4f199aeda06b6322afbb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:37:17.425602 13800 raft_consensus.cc:482] T 00000000000000000000000000000000 P 82216e2b806b4f199aeda06b6322afbb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:37:17.425719 13800 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 82216e2b806b4f199aeda06b6322afbb [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:37:17.426599 13800 raft_consensus.cc:504] T 00000000000000000000000000000000 P 82216e2b806b4f199aeda06b6322afbb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82216e2b806b4f199aeda06b6322afbb" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 35197 } }
I20260929 18:37:17.426932 13800 leader_election.cc:302] T 00000000000000000000000000000000 P 82216e2b806b4f199aeda06b6322afbb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 82216e2b806b4f199aeda06b6322afbb; no voters:
I20260929 18:37:17.427265 13800 leader_election.cc:288] T 00000000000000000000000000000000 P 82216e2b806b4f199aeda06b6322afbb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:37:17.427371 13805 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 82216e2b806b4f199aeda06b6322afbb [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:37:17.427589 13805 raft_consensus.cc:686] T 00000000000000000000000000000000 P 82216e2b806b4f199aeda06b6322afbb [term 1 LEADER]: Becoming Leader. State: Replica: 82216e2b806b4f199aeda06b6322afbb, State: Running, Role: LEADER
I20260929 18:37:17.427996 13805 consensus_queue.cc:237] T 00000000000000000000000000000000 P 82216e2b806b4f199aeda06b6322afbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82216e2b806b4f199aeda06b6322afbb" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 35197 } }
I20260929 18:37:17.428566 13800 sys_catalog.cc:564] T 00000000000000000000000000000000 P 82216e2b806b4f199aeda06b6322afbb [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:37:17.429574 13807 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82216e2b806b4f199aeda06b6322afbb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 82216e2b806b4f199aeda06b6322afbb. Latest consensus state: current_term: 1 leader_uuid: "82216e2b806b4f199aeda06b6322afbb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82216e2b806b4f199aeda06b6322afbb" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 35197 } } }
I20260929 18:37:17.429708 13807 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82216e2b806b4f199aeda06b6322afbb [sys.catalog]: This master's current role is: LEADER
I20260929 18:37:17.430034 13806 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82216e2b806b4f199aeda06b6322afbb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "82216e2b806b4f199aeda06b6322afbb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82216e2b806b4f199aeda06b6322afbb" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 35197 } } }
I20260929 18:37:17.430110 13806 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82216e2b806b4f199aeda06b6322afbb [sys.catalog]: This master's current role is: LEADER
I20260929 18:37:17.430408 13814 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 29 18:37:17 dist-test-slave-j1xs krb5kdc[13719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707037, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.100.62@KRBTEST.COM
I20260929 18:37:17.433624 13814 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:37:17.437516 13802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:17.416434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48644 (local address 127.13.100.62:35197)
0929 18:37:17.416863 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:17.416878 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:17.417799 (+ 921us) server_negotiation.cc:407] Connection header received
0929 18:37:17.420975 (+ 3176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:17.420991 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:17.421274 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:17.421589 (+ 315us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:17.423696 (+ 2107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:17.424609 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:17.426019 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:17.426460 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:17.426630 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:17.431816 (+ 5186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:17.431822 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:17.431838 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:37:17.431869 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:17.433954 (+ 2085us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:17.434615 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:17.434621 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:17.434626 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:17.434759 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:17.435133 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:17.435136 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:17.435138 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:17.435616 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:17.435796 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:17.436439 (+ 643us) server_negotiation.cc:299] Negotiation successful
0929 18:37:17.436705 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":127,"threads_started":1}
I20260929 18:37:17.440389 13814 catalog_manager.cc:1347] Generated new cluster ID: 4c5c66dbc1de4c5e8098e4d067be83f0
I20260929 18:37:17.440495 13814 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:37:17.456321 13814 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:37:17.457569 13814 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:37:17.464540 13814 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 82216e2b806b4f199aeda06b6322afbb: Generated new TSK 0
I20260929 18:37:17.465282 13814 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.100.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.100.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.100.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.1.keytab.
Entry for principal kudu/127.13.100.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.1.keytab.
I20260929 18:37:17.497013 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707036041078-13712-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707036041078-13712-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707036041078-13712-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707036041078-13712-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.100.1:0
--local_ip_for_outbound_sockets=127.13.100.1
--webserver_interface=127.13.100.1
--webserver_port=0
--tserver_master_addrs=127.13.100.62:35197
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.1.keytab
--principal=kudu/127.13.100.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.100.20:38413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707036041078-13712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707036041078-13712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707036041078-13712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:37:17.605671 13826 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:37:17.605931 13826 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:37:17.605974 13826 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:37:17.610105 13826 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:37:17.610185 13826 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:37:17.610244 13826 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.100.1
I20260929 18:37:17.615491 13826 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.100.20:38413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707036041078-13712-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707036041078-13712-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.1.keytab
--principal=kudu/127.13.100.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.100.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707036041078-13712-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.100.1
--webserver_port=0
--tserver_master_addrs=127.13.100.62:35197
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13826
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.100.1
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707036041078-13712-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:37:17.616518 13826 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:37:17.617537 13826 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:37:17.626893 13832 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:37:17.626880 13834 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:37:17.626924 13826 server_base.cc:1034] running on GCE node
W20260929 18:37:17.626880 13831 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:37:17.627552 13826 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:37:17.628114 13826 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:37:17.629362 13826 hybrid_clock.cc:648] HybridClock initialized: now 1790707037629323 us; error 49 us; skew 500 ppm
Sep 29 18:37:17 dist-test-slave-j1xs krb5kdc[13719](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707037, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:37:17.632441 13826 init.cc:377] Logged in from keytab as kudu/127.13.100.1@KRBTEST.COM (short username kudu)
I20260929 18:37:17.633344 13826 webserver.cc:460] Webserver started at http://127.13.100.1:43459/ using document root <none> and password file <none>
I20260929 18:37:17.633961 13826 fs_manager.cc:359] Metadata directory not provided
I20260929 18:37:17.634030 13826 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:37:17.634258 13826 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:37:17.636287 13826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707036041078-13712-0/minicluster-data/ts-0/data/instance:
uuid: "0f29681c86764d30b29baea261b23e55"
format_stamp: "Formatted at 2026-09-29 18:37:17 on dist-test-slave-j1xs"
server_key: "4986fffeb04f7eb169643b6d4e3b2771"
server_key_iv: "bfd443da63de2e5d0e5bba3fb9ac2414"
server_key_version: "encryptionkey@0"
I20260929 18:37:17.636854 13826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707036041078-13712-0/minicluster-data/ts-0/wal/instance:
uuid: "0f29681c86764d30b29baea261b23e55"
format_stamp: "Formatted at 2026-09-29 18:37:17 on dist-test-slave-j1xs"
server_key: "4986fffeb04f7eb169643b6d4e3b2771"
server_key_iv: "bfd443da63de2e5d0e5bba3fb9ac2414"
server_key_version: "encryptionkey@0"
I20260929 18:37:17.641012 13826 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260929 18:37:17.643482 13841 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:37:17.644740 13826 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 18:37:17.644906 13826 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707036041078-13712-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707036041078-13712-0/minicluster-data/ts-0/wal
uuid: "0f29681c86764d30b29baea261b23e55"
format_stamp: "Formatted at 2026-09-29 18:37:17 on dist-test-slave-j1xs"
server_key: "4986fffeb04f7eb169643b6d4e3b2771"
server_key_iv: "bfd443da63de2e5d0e5bba3fb9ac2414"
server_key_version: "encryptionkey@0"
I20260929 18:37:17.645013 13826 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707036041078-13712-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707036041078-13712-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707036041078-13712-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:37:17.658186 13826 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:37:17.661268 13826 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:37:17.661460 13826 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:37:17.662041 13826 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:37:17.662808 13826 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:37:17.662855 13826 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:37:17.662892 13826 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:37:17.662943 13826 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:37:17.671838 13826 rpc_server.cc:307] RPC server started. Bound to: 127.13.100.1:42683
I20260929 18:37:17.671861 13954 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.100.1:42683 every 8 connection(s)
I20260929 18:37:17.672727 13826 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707036041078-13712-0/minicluster-data/ts-0/data/info.pb
I20260929 18:37:17.672933 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 13826
I20260929 18:37:17.673049 13712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707036041078-13712-0/minicluster-data/ts-0/wal/instance
I20260929 18:37:17.673339 13712 external_mini_cluster.cc:1428] Setting key 63acd5d49a65549b434e114764110d5b
WARNING: no policy specified for kudu/127.13.100.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.100.2@KRBTEST.COM" created.
Sep 29 18:37:17 dist-test-slave-j1xs krb5kdc[13719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707037, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.1@KRBTEST.COM for kudu/127.13.100.62@KRBTEST.COM
I20260929 18:37:17.691485 13802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:17.674627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.1:45761 (local address 127.13.100.62:35197)
0929 18:37:17.674785 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:17.674791 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:17.676923 (+ 2132us) server_negotiation.cc:407] Connection header received
0929 18:37:17.678129 (+ 1206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:17.678133 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:17.678188 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:17.678266 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:17.679481 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:17.680021 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:17.681858 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:17.682172 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:17.682259 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:17.686883 (+ 4624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:17.686889 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:17.686896 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:37:17.686933 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:17.689072 (+ 2139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:17.689737 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:17.689739 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:17.689741 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:17.689819 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:17.690284 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:17.690286 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:17.690288 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:17.690468 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:17.690548 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:17.691243 (+ 695us) server_negotiation.cc:299] Negotiation successful
0929 18:37:17.691358 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 18:37:17.692931 13957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:17.675105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:35197 (local address 127.13.100.1:45761)
0929 18:37:17.676751 (+ 1646us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:17.676809 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:17.677873 (+ 1064us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:17.678428 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:17.678435 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:17.678857 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:37:17.679339 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:17.679352 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:17.680190 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:17.680197 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:17.681715 (+ 1518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:17.681727 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:17.682335 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:17.682340 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:17.682574 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:17.683299 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:37:17.683315 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:37:17.686698 (+ 3383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:37:17.689198 (+ 2500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:17.689207 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:17.689227 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:17.689624 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:17.689961 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:17.689968 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:17.689973 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:17.690172 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:17.690581 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:37:17.690588 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:37:17.690951 (+ 363us) client_negotiation.cc:769] Sending connection context
0929 18:37:17.691191 (+ 240us) client_negotiation.cc:240] Negotiation successful
0929 18:37:17.691593 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1409,"thread_start_us":689,"threads_started":1}
I20260929 18:37:17.694334 13955 heartbeater.cc:347] Connected to a master server at 127.13.100.62:35197
I20260929 18:37:17.694583 13955 heartbeater.cc:464] Registering TS with master...
I20260929 18:37:17.695075 13955 heartbeater.cc:511] Master 127.13.100.62:35197 requested a full tablet report, sending...
I20260929 18:37:17.696753 13764 ts_manager.cc:194] Registered new tserver with Master: 0f29681c86764d30b29baea261b23e55 (127.13.100.1:42683)
I20260929 18:37:17.698045 13764 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.100.1@KRBTEST.COM'} at 127.13.100.1:45761
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.100.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.2.keytab.
Entry for principal kudu/127.13.100.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.2.keytab.
I20260929 18:37:17.704923 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707036041078-13712-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707036041078-13712-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707036041078-13712-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707036041078-13712-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.100.2:0
--local_ip_for_outbound_sockets=127.13.100.2
--webserver_interface=127.13.100.2
--webserver_port=0
--tserver_master_addrs=127.13.100.62:35197
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.2.keytab
--principal=kudu/127.13.100.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.100.20:38413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707036041078-13712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707036041078-13712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707036041078-13712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:37:17.818064 13960 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:37:17.818341 13960 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:37:17.818384 13960 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:37:17.822551 13960 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:37:17.822610 13960 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:37:17.822724 13960 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.100.2
I20260929 18:37:17.827886 13960 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.100.20:38413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707036041078-13712-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707036041078-13712-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.2.keytab
--principal=kudu/127.13.100.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.100.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707036041078-13712-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.100.2
--webserver_port=0
--tserver_master_addrs=127.13.100.62:35197
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13960
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.100.2
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707036041078-13712-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:37:17.829017 13960 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:37:17.830173 13960 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:37:17.839707 13966 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:37:17.839707 13968 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:37:17.839761 13965 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:37:17.840050 13960 server_base.cc:1034] running on GCE node
I20260929 18:37:17.840441 13960 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:37:17.841101 13960 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:37:17.842295 13960 hybrid_clock.cc:648] HybridClock initialized: now 1790707037842274 us; error 36 us; skew 500 ppm
Sep 29 18:37:17 dist-test-slave-j1xs krb5kdc[13719](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707037, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:37:17.844879 13960 init.cc:377] Logged in from keytab as kudu/127.13.100.2@KRBTEST.COM (short username kudu)
I20260929 18:37:17.845822 13960 webserver.cc:460] Webserver started at http://127.13.100.2:45883/ using document root <none> and password file <none>
I20260929 18:37:17.846413 13960 fs_manager.cc:359] Metadata directory not provided
I20260929 18:37:17.846481 13960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:37:17.846654 13960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:37:17.848748 13960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707036041078-13712-0/minicluster-data/ts-1/data/instance:
uuid: "1474a18621ff40269d49362ad73c5fe2"
format_stamp: "Formatted at 2026-09-29 18:37:17 on dist-test-slave-j1xs"
server_key: "c04966bfa824f31670b4fff7aafd6f68"
server_key_iv: "d3dd327649a15f71c3564830f86e6d7e"
server_key_version: "encryptionkey@0"
I20260929 18:37:17.849283 13960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707036041078-13712-0/minicluster-data/ts-1/wal/instance:
uuid: "1474a18621ff40269d49362ad73c5fe2"
format_stamp: "Formatted at 2026-09-29 18:37:17 on dist-test-slave-j1xs"
server_key: "c04966bfa824f31670b4fff7aafd6f68"
server_key_iv: "d3dd327649a15f71c3564830f86e6d7e"
server_key_version: "encryptionkey@0"
I20260929 18:37:17.853127 13960 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 18:37:17.855533 13975 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:37:17.856731 13960 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 18:37:17.856827 13960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707036041078-13712-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707036041078-13712-0/minicluster-data/ts-1/wal
uuid: "1474a18621ff40269d49362ad73c5fe2"
format_stamp: "Formatted at 2026-09-29 18:37:17 on dist-test-slave-j1xs"
server_key: "c04966bfa824f31670b4fff7aafd6f68"
server_key_iv: "d3dd327649a15f71c3564830f86e6d7e"
server_key_version: "encryptionkey@0"
I20260929 18:37:17.856909 13960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707036041078-13712-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707036041078-13712-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707036041078-13712-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:37:17.879473 13960 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:37:17.882560 13960 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:37:17.882712 13960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:37:17.883242 13960 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:37:17.883975 13960 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:37:17.884020 13960 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:37:17.884058 13960 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:37:17.884079 13960 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:37:17.894124 13960 rpc_server.cc:307] RPC server started. Bound to: 127.13.100.2:38163
I20260929 18:37:17.894230 14088 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.100.2:38163 every 8 connection(s)
I20260929 18:37:17.895010 13960 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707036041078-13712-0/minicluster-data/ts-1/data/info.pb
I20260929 18:37:17.901659 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 13960
I20260929 18:37:17.901796 13712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707036041078-13712-0/minicluster-data/ts-1/wal/instance
I20260929 18:37:17.902096 13712 external_mini_cluster.cc:1428] Setting key ea634c95820ed93c5a9ed5dd80d74542
Sep 29 18:37:17 dist-test-slave-j1xs krb5kdc[13719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707037, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.2@KRBTEST.COM for kudu/127.13.100.62@KRBTEST.COM
I20260929 18:37:17.908732 13802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:17.896725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.2:59025 (local address 127.13.100.62:35197)
0929 18:37:17.896856 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:17.896860 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:17.897633 (+ 773us) server_negotiation.cc:407] Connection header received
0929 18:37:17.898945 (+ 1312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:17.898948 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:17.899005 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:17.899105 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:17.900199 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:17.900718 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:17.901873 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:17.902237 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:17.902356 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:17.905060 (+ 2704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:17.905063 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:17.905064 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:37:17.905087 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:17.906548 (+ 1461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:17.907075 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:17.907082 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:17.907086 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:17.907186 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:17.907505 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:17.907509 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:17.907511 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:17.907781 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:17.907911 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:17.908348 (+ 437us) server_negotiation.cc:299] Negotiation successful
0929 18:37:17.908486 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 18:37:17.909312 14091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:17.897086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:35197 (local address 127.13.100.2:59025)
0929 18:37:17.897500 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:17.897538 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:17.898738 (+ 1200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:17.899287 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:17.899293 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:17.899621 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:37:17.900061 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:17.900074 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:17.900838 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:17.900841 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:17.901730 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:17.901739 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:17.902383 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:17.902386 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:17.902528 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:17.903173 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:37:17.903189 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:37:17.904890 (+ 1701us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:37:17.906667 (+ 1777us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:17.906674 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:17.906687 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:17.906933 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:17.907288 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:17.907291 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:17.907292 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:17.907413 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:17.907899 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:37:17.907903 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:37:17.908111 (+ 208us) client_negotiation.cc:769] Sending connection context
0929 18:37:17.908260 (+ 149us) client_negotiation.cc:240] Negotiation successful
0929 18:37:17.908486 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":110,"threads_started":1}
I20260929 18:37:17.910441 14089 heartbeater.cc:347] Connected to a master server at 127.13.100.62:35197
I20260929 18:37:17.910681 14089 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.13.100.3@KRBTEST.COM; defaulting to no policy
I20260929 18:37:17.911140 14089 heartbeater.cc:511] Master 127.13.100.62:35197 requested a full tablet report, sending...
I20260929 18:37:17.912096 13764 ts_manager.cc:194] Registered new tserver with Master: 1474a18621ff40269d49362ad73c5fe2 (127.13.100.2:38163)
I20260929 18:37:17.912669 13764 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.100.2@KRBTEST.COM'} at 127.13.100.2:59025
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.100.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.100.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.3.keytab.
Entry for principal kudu/127.13.100.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.3.keytab.
I20260929 18:37:17.929103 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707036041078-13712-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707036041078-13712-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707036041078-13712-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707036041078-13712-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.100.3:0
--local_ip_for_outbound_sockets=127.13.100.3
--webserver_interface=127.13.100.3
--webserver_port=0
--tserver_master_addrs=127.13.100.62:35197
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.3.keytab
--principal=kudu/127.13.100.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.100.20:38413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707036041078-13712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707036041078-13712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707036041078-13712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:37:18.039115 14094 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:37:18.039382 14094 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:37:18.039453 14094 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:37:18.043396 14094 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:37:18.043470 14094 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:37:18.043553 14094 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.100.3
I20260929 18:37:18.048730 14094 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.100.20:38413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707036041078-13712-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707036041078-13712-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.3.keytab
--principal=kudu/127.13.100.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.100.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707036041078-13712-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.100.3
--webserver_port=0
--tserver_master_addrs=127.13.100.62:35197
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14094
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.100.3
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707036041078-13712-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:37:18.049787 14094 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:37:18.050794 14094 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:37:18.060485 14102 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:37:18.060492 14100 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:37:18.060508 14099 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:37:18.060770 14094 server_base.cc:1034] running on GCE node
I20260929 18:37:18.061280 14094 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:37:18.061981 14094 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:37:18.063199 14094 hybrid_clock.cc:648] HybridClock initialized: now 1790707038063163 us; error 47 us; skew 500 ppm
Sep 29 18:37:18 dist-test-slave-j1xs krb5kdc[13719](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707038, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:37:18.065999 14094 init.cc:377] Logged in from keytab as kudu/127.13.100.3@KRBTEST.COM (short username kudu)
I20260929 18:37:18.066840 14094 webserver.cc:460] Webserver started at http://127.13.100.3:32999/ using document root <none> and password file <none>
I20260929 18:37:18.067399 14094 fs_manager.cc:359] Metadata directory not provided
I20260929 18:37:18.067487 14094 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:37:18.067713 14094 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:37:18.069730 14094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707036041078-13712-0/minicluster-data/ts-2/data/instance:
uuid: "43758dfc94174071a7b1cd3818bbfc6f"
format_stamp: "Formatted at 2026-09-29 18:37:18 on dist-test-slave-j1xs"
server_key: "ad7a7942292932dba619cd1523821fe8"
server_key_iv: "028e2a5429b506427af046c59bd6298e"
server_key_version: "encryptionkey@0"
I20260929 18:37:18.070353 14094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707036041078-13712-0/minicluster-data/ts-2/wal/instance:
uuid: "43758dfc94174071a7b1cd3818bbfc6f"
format_stamp: "Formatted at 2026-09-29 18:37:18 on dist-test-slave-j1xs"
server_key: "ad7a7942292932dba619cd1523821fe8"
server_key_iv: "028e2a5429b506427af046c59bd6298e"
server_key_version: "encryptionkey@0"
I20260929 18:37:18.074100 14094 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260929 18:37:18.076704 14109 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:37:18.077982 14094 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:37:18.078104 14094 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707036041078-13712-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707036041078-13712-0/minicluster-data/ts-2/wal
uuid: "43758dfc94174071a7b1cd3818bbfc6f"
format_stamp: "Formatted at 2026-09-29 18:37:18 on dist-test-slave-j1xs"
server_key: "ad7a7942292932dba619cd1523821fe8"
server_key_iv: "028e2a5429b506427af046c59bd6298e"
server_key_version: "encryptionkey@0"
I20260929 18:37:18.078212 14094 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707036041078-13712-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707036041078-13712-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707036041078-13712-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:37:18.091104 14094 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:37:18.094338 14094 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:37:18.094527 14094 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:37:18.095124 14094 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:37:18.095901 14094 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:37:18.095968 14094 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:37:18.096032 14094 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:37:18.096078 14094 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:37:18.104851 14094 rpc_server.cc:307] RPC server started. Bound to: 127.13.100.3:46549
I20260929 18:37:18.104902 14222 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.100.3:46549 every 8 connection(s)
I20260929 18:37:18.105717 14094 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707036041078-13712-0/minicluster-data/ts-2/data/info.pb
Sep 29 18:37:18 dist-test-slave-j1xs krb5kdc[13719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707038, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.3@KRBTEST.COM for kudu/127.13.100.62@KRBTEST.COM
I20260929 18:37:18.115206 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 14094
I20260929 18:37:18.115309 13712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707036041078-13712-0/minicluster-data/ts-2/wal/instance
I20260929 18:37:18.115628 13712 external_mini_cluster.cc:1428] Setting key 87505368030318f18c33e73f09a835c2
I20260929 18:37:18.118901 13802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:18.107643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.3:51131 (local address 127.13.100.62:35197)
0929 18:37:18.107824 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:18.107829 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:18.108571 (+ 742us) server_negotiation.cc:407] Connection header received
0929 18:37:18.109790 (+ 1219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:18.109793 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:18.109837 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:18.109979 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:18.111036 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:18.111568 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:18.112623 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:18.112846 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:18.112928 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:18.115254 (+ 2326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:18.115259 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:18.115263 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:37:18.115294 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:18.116889 (+ 1595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:18.117390 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:18.117394 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:18.117396 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:18.117479 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:18.117805 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:18.117810 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:18.117812 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:18.118052 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:18.118161 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:18.118620 (+ 459us) server_negotiation.cc:299] Negotiation successful
0929 18:37:18.118735 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 18:37:18.119699 14225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:18.107993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:35197 (local address 127.13.100.3:51131)
0929 18:37:18.108427 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:18.108466 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:18.109606 (+ 1140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:18.110116 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:18.110121 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:18.110455 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:37:18.110907 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:18.110919 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:18.111679 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:18.111681 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:18.112513 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:18.112521 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:18.112969 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:18.112971 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:18.113108 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:18.113634 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:37:18.113649 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:37:18.115090 (+ 1441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:37:18.117020 (+ 1930us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:18.117025 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:18.117039 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:18.117287 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:18.117577 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:18.117580 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:18.117582 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:18.117712 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:18.118157 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:37:18.118163 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:37:18.118393 (+ 230us) client_negotiation.cc:769] Sending connection context
0929 18:37:18.118570 (+ 177us) client_negotiation.cc:240] Negotiation successful
0929 18:37:18.118862 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":117,"threads_started":1}
I20260929 18:37:18.120767 14223 heartbeater.cc:347] Connected to a master server at 127.13.100.62:35197
I20260929 18:37:18.120966 14223 heartbeater.cc:464] Registering TS with master...
I20260929 18:37:18.121423 14223 heartbeater.cc:511] Master 127.13.100.62:35197 requested a full tablet report, sending...
I20260929 18:37:18.122390 13764 ts_manager.cc:194] Registered new tserver with Master: 43758dfc94174071a7b1cd3818bbfc6f (127.13.100.3:46549)
I20260929 18:37:18.122934 13764 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.100.3@KRBTEST.COM'} at 127.13.100.3:51131
I20260929 18:37:18.129779 13712 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 18:37:18 dist-test-slave-j1xs krb5kdc[13719](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707038, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 18:37:18 dist-test-slave-j1xs krb5kdc[13719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707038, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.13.100.1@KRBTEST.COM
I20260929 18:37:18.155620 14230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:18.146121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35364 (local address 127.13.100.1:42683)
0929 18:37:18.146461 (+ 340us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:18.146476 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:18.146501 (+ 25us) server_negotiation.cc:407] Connection header received
0929 18:37:18.146611 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:18.146618 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:18.146820 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:18.146974 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:18.147657 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:18.148504 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:18.149350 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:18.149774 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:18.149886 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:18.152384 (+ 2498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:18.152392 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:18.152405 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:37:18.152429 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:18.153887 (+ 1458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:18.154469 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:18.154474 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:18.154475 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:18.154555 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:18.154787 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:18.154789 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:18.154791 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:18.155016 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:18.155184 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:18.155352 (+ 168us) server_negotiation.cc:299] Negotiation successful
0929 18:37:18.155472 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":220,"thread_start_us":106,"threads_started":1}
W20260929 18:37:18.156575 13869 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35364
Sep 29 18:37:18 dist-test-slave-j1xs krb5kdc[13719](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707038, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 18:37:18 dist-test-slave-j1xs krb5kdc[13719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707038, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.100.1@KRBTEST.COM
I20260929 18:37:18.185340 14230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:18.176724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35376 (local address 127.13.100.1:42683)
0929 18:37:18.176952 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:18.176957 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:18.176974 (+ 17us) server_negotiation.cc:407] Connection header received
0929 18:37:18.177027 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:18.177030 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:18.177077 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:18.177171 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:18.178047 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:18.178583 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:18.179407 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:18.179630 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:18.179704 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:18.182245 (+ 2541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:18.182248 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:18.182251 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:37:18.182276 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:18.183821 (+ 1545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:18.184231 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:18.184234 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:18.184235 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:18.184303 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:18.184540 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:18.184542 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:18.184543 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:18.184694 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:18.184793 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:18.185071 (+ 278us) server_negotiation.cc:299] Negotiation successful
0929 18:37:18.185207 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260929 18:37:18.186486 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 13826
I20260929 18:37:18.192353 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 13960
I20260929 18:37:18.197854 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 14094
I20260929 18:37:18.202996 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 13732
2026-09-29T18:37:18Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2157 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707036041078-13712-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:37:18 dist-test-slave-j1xs krb5kdc[14237](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:37:18 dist-test-slave-j1xs krb5kdc[14237](info): set up 2 sockets
Sep 29 18:37:18 dist-test-slave-j1xs krb5kdc[14237](info): commencing operation
krb5kdc: starting...
W20260929 18:37:19.249526 13712 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707036041078-13712-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707036041078-13712-0/krb5kdc/test-user.keytab.
Sep 29 18:37:19 dist-test-slave-j1xs krb5kdc[14237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707039, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:37:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:37:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.100.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.100.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.100.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.62.keytab.
Entry for principal kudu/127.13.100.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.62.keytab.
I20260929 18:37:19.366946 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707036041078-13712-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707036041078-13712-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707036041078-13712-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.100.62:45687
--webserver_interface=127.13.100.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.62.keytab
--principal=kudu/127.13.100.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.100.20:44345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.100.62:45687
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707036041078-13712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707036041078-13712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707036041078-13712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:37:19.476227 14250 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:37:19.476493 14250 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:37:19.476533 14250 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:37:19.480401 14250 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:37:19.480458 14250 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:37:19.480476 14250 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:37:19.480492 14250 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:37:19.480508 14250 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:37:19.485829 14250 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.100.20:44345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707036041078-13712-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707036041078-13712-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.100.62:45687
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.62.keytab
--principal=kudu/127.13.100.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.100.62:45687
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.100.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14250
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707036041078-13712-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:37:19.486848 14250 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:37:19.487797 14250 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:37:19.496291 14256 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:37:19.496256 14255 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:37:19.496397 14250 server_base.cc:1034] running on GCE node
W20260929 18:37:19.496256 14258 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:37:19.497054 14250 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:37:19.498188 14250 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:37:19.499409 14250 hybrid_clock.cc:648] HybridClock initialized: now 1790707039499380 us; error 40 us; skew 500 ppm
Sep 29 18:37:19 dist-test-slave-j1xs krb5kdc[14237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707039, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:37:19.502321 14250 init.cc:377] Logged in from keytab as kudu/127.13.100.62@KRBTEST.COM (short username kudu)
I20260929 18:37:19.503208 14250 webserver.cc:460] Webserver started at http://127.13.100.62:45183/ using document root <none> and password file <none>
I20260929 18:37:19.503714 14250 fs_manager.cc:359] Metadata directory not provided
I20260929 18:37:19.503772 14250 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:37:19.503932 14250 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:37:19.505877 14250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707036041078-13712-0/minicluster-data/master-0/data/instance:
uuid: "d0a1eb648ca440d28d811e7d6d3e0c36"
format_stamp: "Formatted at 2026-09-29 18:37:19 on dist-test-slave-j1xs"
server_key: "7de23e1e407dfe784a978044f205759b"
server_key_iv: "9f62172554ccd2d4f91a46258cffc271"
server_key_version: "encryptionkey@0"
I20260929 18:37:19.506453 14250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707036041078-13712-0/minicluster-data/master-0/wal/instance:
uuid: "d0a1eb648ca440d28d811e7d6d3e0c36"
format_stamp: "Formatted at 2026-09-29 18:37:19 on dist-test-slave-j1xs"
server_key: "7de23e1e407dfe784a978044f205759b"
server_key_iv: "9f62172554ccd2d4f91a46258cffc271"
server_key_version: "encryptionkey@0"
I20260929 18:37:19.509958 14250 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:37:19.512240 14265 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:37:19.513536 14250 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:37:19.513669 14250 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707036041078-13712-0/minicluster-data/master-0/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707036041078-13712-0/minicluster-data/master-0/wal
uuid: "d0a1eb648ca440d28d811e7d6d3e0c36"
format_stamp: "Formatted at 2026-09-29 18:37:19 on dist-test-slave-j1xs"
server_key: "7de23e1e407dfe784a978044f205759b"
server_key_iv: "9f62172554ccd2d4f91a46258cffc271"
server_key_version: "encryptionkey@0"
I20260929 18:37:19.513779 14250 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707036041078-13712-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707036041078-13712-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707036041078-13712-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:37:19.543387 14250 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:37:19.546567 14250 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:37:19.546787 14250 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:37:19.553814 14250 rpc_server.cc:307] RPC server started. Bound to: 127.13.100.62:45687
I20260929 18:37:19.553831 14317 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.100.62:45687 every 8 connection(s)
I20260929 18:37:19.554831 14250 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
I20260929 18:37:19.557727 14318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:37:19.563216 14318 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0a1eb648ca440d28d811e7d6d3e0c36: Bootstrap starting.
I20260929 18:37:19.563659 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 14250
I20260929 18:37:19.563804 13712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707036041078-13712-0/minicluster-data/master-0/wal/instance
I20260929 18:37:19.564071 13712 external_mini_cluster.cc:1428] Setting key 57c814346a57d45260bdaa6ed82f5fb1
I20260929 18:37:19.565951 14318 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d0a1eb648ca440d28d811e7d6d3e0c36: Neither blocks nor log segments found. Creating new log.
I20260929 18:37:19.566702 14318 log.cc:824] T 00000000000000000000000000000000 P d0a1eb648ca440d28d811e7d6d3e0c36: Log is configured to *not* fsync() on all Append() calls
I20260929 18:37:19.569000 14318 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0a1eb648ca440d28d811e7d6d3e0c36: No bootstrap required, opened a new log
I20260929 18:37:19.572996 14318 raft_consensus.cc:348] T 00000000000000000000000000000000 P d0a1eb648ca440d28d811e7d6d3e0c36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0a1eb648ca440d28d811e7d6d3e0c36" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 45687 } }
Sep 29 18:37:19 dist-test-slave-j1xs krb5kdc[14237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707039, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.100.62@KRBTEST.COM
I20260929 18:37:19.573248 14318 raft_consensus.cc:374] T 00000000000000000000000000000000 P d0a1eb648ca440d28d811e7d6d3e0c36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:37:19.573334 14318 raft_consensus.cc:729] T 00000000000000000000000000000000 P d0a1eb648ca440d28d811e7d6d3e0c36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0a1eb648ca440d28d811e7d6d3e0c36, State: Initialized, Role: FOLLOWER
I20260929 18:37:19.573840 14318 consensus_queue.cc:260] T 00000000000000000000000000000000 P d0a1eb648ca440d28d811e7d6d3e0c36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0a1eb648ca440d28d811e7d6d3e0c36" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 45687 } }
I20260929 18:37:19.574043 14318 raft_consensus.cc:388] T 00000000000000000000000000000000 P d0a1eb648ca440d28d811e7d6d3e0c36 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:37:19.574126 14318 raft_consensus.cc:482] T 00000000000000000000000000000000 P d0a1eb648ca440d28d811e7d6d3e0c36 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:37:19.574220 14318 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d0a1eb648ca440d28d811e7d6d3e0c36 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:37:19.575292 14318 raft_consensus.cc:504] T 00000000000000000000000000000000 P d0a1eb648ca440d28d811e7d6d3e0c36 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0a1eb648ca440d28d811e7d6d3e0c36" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 45687 } }
I20260929 18:37:19.575632 14318 leader_election.cc:302] T 00000000000000000000000000000000 P d0a1eb648ca440d28d811e7d6d3e0c36 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d0a1eb648ca440d28d811e7d6d3e0c36; no voters:
I20260929 18:37:19.575971 14318 leader_election.cc:288] T 00000000000000000000000000000000 P d0a1eb648ca440d28d811e7d6d3e0c36 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:37:19.576018 14323 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d0a1eb648ca440d28d811e7d6d3e0c36 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:37:19.576296 14323 raft_consensus.cc:686] T 00000000000000000000000000000000 P d0a1eb648ca440d28d811e7d6d3e0c36 [term 1 LEADER]: Becoming Leader. State: Replica: d0a1eb648ca440d28d811e7d6d3e0c36, State: Running, Role: LEADER
I20260929 18:37:19.576581 14323 consensus_queue.cc:237] T 00000000000000000000000000000000 P d0a1eb648ca440d28d811e7d6d3e0c36 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0a1eb648ca440d28d811e7d6d3e0c36" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 45687 } }
I20260929 18:37:19.577229 14318 sys_catalog.cc:564] T 00000000000000000000000000000000 P d0a1eb648ca440d28d811e7d6d3e0c36 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:37:19.577714 14325 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0a1eb648ca440d28d811e7d6d3e0c36 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d0a1eb648ca440d28d811e7d6d3e0c36. Latest consensus state: current_term: 1 leader_uuid: "d0a1eb648ca440d28d811e7d6d3e0c36" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0a1eb648ca440d28d811e7d6d3e0c36" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 45687 } } }
I20260929 18:37:19.577817 14325 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0a1eb648ca440d28d811e7d6d3e0c36 [sys.catalog]: This master's current role is: LEADER
I20260929 18:37:19.578534 14324 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0a1eb648ca440d28d811e7d6d3e0c36 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d0a1eb648ca440d28d811e7d6d3e0c36" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0a1eb648ca440d28d811e7d6d3e0c36" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 45687 } } }
I20260929 18:37:19.578634 14324 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0a1eb648ca440d28d811e7d6d3e0c36 [sys.catalog]: This master's current role is: LEADER
I20260929 18:37:19.579064 14321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:19.565514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39842 (local address 127.13.100.62:45687)
0929 18:37:19.565955 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:19.565971 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:19.566026 (+ 55us) server_negotiation.cc:407] Connection header received
0929 18:37:19.566693 (+ 667us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:19.566710 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:19.567079 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:19.567358 (+ 279us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:19.568176 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:19.569583 (+ 1407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:19.570511 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:19.571212 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:19.571365 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:19.573874 (+ 2509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:19.573879 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:19.573931 (+ 52us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:37:19.573967 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:19.575777 (+ 1810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:19.576302 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:19.576308 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:19.576313 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:19.576442 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:19.576763 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:19.576766 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:19.576768 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:19.577256 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:19.577442 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:19.577863 (+ 421us) server_negotiation.cc:299] Negotiation successful
0929 18:37:19.578201 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":135,"threads_started":1}
I20260929 18:37:19.579722 14328 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:37:19.583102 14328 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:37:19.588425 14328 catalog_manager.cc:1347] Generated new cluster ID: 1f1de47223d545838b11f026d70d0e85
I20260929 18:37:19.588487 14328 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:37:19.599064 14328 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:37:19.600229 14328 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:37:19.610855 14328 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d0a1eb648ca440d28d811e7d6d3e0c36: Generated new TSK 0
I20260929 18:37:19.611716 14328 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.100.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.100.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.100.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.1.keytab.
Entry for principal kudu/127.13.100.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.1.keytab.
I20260929 18:37:19.649998 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707036041078-13712-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707036041078-13712-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707036041078-13712-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707036041078-13712-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.100.1:0
--local_ip_for_outbound_sockets=127.13.100.1
--webserver_interface=127.13.100.1
--webserver_port=0
--tserver_master_addrs=127.13.100.62:45687
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.1.keytab
--principal=kudu/127.13.100.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.100.20:44345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707036041078-13712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707036041078-13712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707036041078-13712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:37:19.762821 14344 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:37:19.763069 14344 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:37:19.763113 14344 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:37:19.767128 14344 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:37:19.767185 14344 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:37:19.767243 14344 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.100.1
I20260929 18:37:19.772281 14344 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.100.20:44345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707036041078-13712-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707036041078-13712-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.1.keytab
--principal=kudu/127.13.100.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.100.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707036041078-13712-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.100.1
--webserver_port=0
--tserver_master_addrs=127.13.100.62:45687
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14344
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.100.1
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707036041078-13712-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:37:19.773208 14344 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:37:19.774147 14344 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:37:19.783834 14349 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:37:19.783941 14352 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:37:19.783831 14350 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:37:19.783867 14344 server_base.cc:1034] running on GCE node
I20260929 18:37:19.784497 14344 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:37:19.785066 14344 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:37:19.786235 14344 hybrid_clock.cc:648] HybridClock initialized: now 1790707039786219 us; error 35 us; skew 500 ppm
Sep 29 18:37:19 dist-test-slave-j1xs krb5kdc[14237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707039, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:37:19.789005 14344 init.cc:377] Logged in from keytab as kudu/127.13.100.1@KRBTEST.COM (short username kudu)
I20260929 18:37:19.789805 14344 webserver.cc:460] Webserver started at http://127.13.100.1:45041/ using document root <none> and password file <none>
I20260929 18:37:19.790359 14344 fs_manager.cc:359] Metadata directory not provided
I20260929 18:37:19.790418 14344 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:37:19.790634 14344 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:37:19.792582 14344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707036041078-13712-0/minicluster-data/ts-0/data/instance:
uuid: "02a35256f86d41908c390a07804ef794"
format_stamp: "Formatted at 2026-09-29 18:37:19 on dist-test-slave-j1xs"
server_key: "901eabec2ca71985cb0c5899b7f4b72b"
server_key_iv: "bc79c36d75ca43255bdcc7d571e9802a"
server_key_version: "encryptionkey@0"
I20260929 18:37:19.793223 14344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707036041078-13712-0/minicluster-data/ts-0/wal/instance:
uuid: "02a35256f86d41908c390a07804ef794"
format_stamp: "Formatted at 2026-09-29 18:37:19 on dist-test-slave-j1xs"
server_key: "901eabec2ca71985cb0c5899b7f4b72b"
server_key_iv: "bc79c36d75ca43255bdcc7d571e9802a"
server_key_version: "encryptionkey@0"
I20260929 18:37:19.796761 14344 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260929 18:37:19.799247 14359 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:37:19.800414 14344 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 18:37:19.800540 14344 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707036041078-13712-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707036041078-13712-0/minicluster-data/ts-0/wal
uuid: "02a35256f86d41908c390a07804ef794"
format_stamp: "Formatted at 2026-09-29 18:37:19 on dist-test-slave-j1xs"
server_key: "901eabec2ca71985cb0c5899b7f4b72b"
server_key_iv: "bc79c36d75ca43255bdcc7d571e9802a"
server_key_version: "encryptionkey@0"
I20260929 18:37:19.800657 14344 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707036041078-13712-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707036041078-13712-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707036041078-13712-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:37:19.823972 14344 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:37:19.826992 14344 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:37:19.827193 14344 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:37:19.827957 14344 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:37:19.828878 14344 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:37:19.828922 14344 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:37:19.828997 14344 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:37:19.829026 14344 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:37:19.837797 14344 rpc_server.cc:307] RPC server started. Bound to: 127.13.100.1:34283
I20260929 18:37:19.837822 14472 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.100.1:34283 every 8 connection(s)
I20260929 18:37:19.838702 14344 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707036041078-13712-0/minicluster-data/ts-0/data/info.pb
I20260929 18:37:19.846676 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 14344
I20260929 18:37:19.846765 13712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707036041078-13712-0/minicluster-data/ts-0/wal/instance
I20260929 18:37:19.847012 13712 external_mini_cluster.cc:1428] Setting key ba3481c6068d33afe12672b39dde9d01
Sep 29 18:37:19 dist-test-slave-j1xs krb5kdc[14237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707039, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.1@KRBTEST.COM for kudu/127.13.100.62@KRBTEST.COM
I20260929 18:37:19.851898 14321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:19.840444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.1:34351 (local address 127.13.100.62:45687)
0929 18:37:19.840604 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:19.840609 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:19.841421 (+ 812us) server_negotiation.cc:407] Connection header received
0929 18:37:19.842301 (+ 880us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:19.842304 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:19.842348 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:19.842421 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:19.843534 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:19.844054 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:19.845106 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:19.845329 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:19.845411 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:19.847957 (+ 2546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:19.847960 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:19.847963 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:37:19.847986 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:19.849631 (+ 1645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:19.850262 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:19.850269 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:19.850272 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:19.850373 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:19.850739 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:19.850745 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:19.850748 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:19.850992 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:19.851098 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:19.851681 (+ 583us) server_negotiation.cc:299] Negotiation successful
0929 18:37:19.851782 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 18:37:19.853111 14475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:19.840827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:45687 (local address 127.13.100.1:34351)
0929 18:37:19.841273 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:19.841312 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:19.842118 (+ 806us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:19.842569 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:19.842575 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:19.842912 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:37:19.843373 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:19.843386 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:19.844172 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:19.844174 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:19.844998 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:19.845005 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:19.845454 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:19.845457 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:19.845587 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:19.846229 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:37:19.846244 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:37:19.847807 (+ 1563us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:37:19.849756 (+ 1949us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:19.849765 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:19.849782 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:19.850154 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:19.850481 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:19.850484 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:19.850485 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:19.850633 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:19.851102 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:37:19.851108 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:37:19.851423 (+ 315us) client_negotiation.cc:769] Sending connection context
0929 18:37:19.851632 (+ 209us) client_negotiation.cc:240] Negotiation successful
0929 18:37:19.851972 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":115,"threads_started":1}
I20260929 18:37:19.854344 14473 heartbeater.cc:347] Connected to a master server at 127.13.100.62:45687
I20260929 18:37:19.854601 14473 heartbeater.cc:464] Registering TS with master...
I20260929 18:37:19.855182 14473 heartbeater.cc:511] Master 127.13.100.62:45687 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.13.100.2@KRBTEST.COM; defaulting to no policy
I20260929 18:37:19.856721 14282 ts_manager.cc:194] Registered new tserver with Master: 02a35256f86d41908c390a07804ef794 (127.13.100.1:34283)
I20260929 18:37:19.857923 14282 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.100.1@KRBTEST.COM'} at 127.13.100.1:34351
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.100.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.100.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.2.keytab.
Entry for principal kudu/127.13.100.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.2.keytab.
I20260929 18:37:19.874576 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707036041078-13712-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707036041078-13712-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707036041078-13712-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707036041078-13712-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.100.2:0
--local_ip_for_outbound_sockets=127.13.100.2
--webserver_interface=127.13.100.2
--webserver_port=0
--tserver_master_addrs=127.13.100.62:45687
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.2.keytab
--principal=kudu/127.13.100.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.100.20:44345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707036041078-13712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707036041078-13712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707036041078-13712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:37:19.982620 14478 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:37:19.982851 14478 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:37:19.982893 14478 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:37:19.986768 14478 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:37:19.986817 14478 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:37:19.986904 14478 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.100.2
I20260929 18:37:19.992022 14478 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.100.20:44345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707036041078-13712-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707036041078-13712-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.2.keytab
--principal=kudu/127.13.100.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.100.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707036041078-13712-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.100.2
--webserver_port=0
--tserver_master_addrs=127.13.100.62:45687
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14478
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.100.2
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707036041078-13712-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:37:19.992981 14478 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:37:19.994079 14478 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:37:20.003391 14483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:37:20.003459 14478 server_base.cc:1034] running on GCE node
W20260929 18:37:20.003386 14486 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:37:20.003386 14484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:37:20.003953 14478 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:37:20.004628 14478 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:37:20.005784 14478 hybrid_clock.cc:648] HybridClock initialized: now 1790707040005767 us; error 27 us; skew 500 ppm
Sep 29 18:37:20 dist-test-slave-j1xs krb5kdc[14237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707040, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:37:20.008553 14478 init.cc:377] Logged in from keytab as kudu/127.13.100.2@KRBTEST.COM (short username kudu)
I20260929 18:37:20.009330 14478 webserver.cc:460] Webserver started at http://127.13.100.2:35567/ using document root <none> and password file <none>
I20260929 18:37:20.009858 14478 fs_manager.cc:359] Metadata directory not provided
I20260929 18:37:20.009979 14478 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:37:20.010152 14478 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:37:20.012099 14478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707036041078-13712-0/minicluster-data/ts-1/data/instance:
uuid: "e982a14ebab146c283af8dc996db9d72"
format_stamp: "Formatted at 2026-09-29 18:37:20 on dist-test-slave-j1xs"
server_key: "7cc2adcf6245e0855bd09ded1f48fbf4"
server_key_iv: "e4c8f23e8789e50b9395037f187a7552"
server_key_version: "encryptionkey@0"
I20260929 18:37:20.012616 14478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707036041078-13712-0/minicluster-data/ts-1/wal/instance:
uuid: "e982a14ebab146c283af8dc996db9d72"
format_stamp: "Formatted at 2026-09-29 18:37:20 on dist-test-slave-j1xs"
server_key: "7cc2adcf6245e0855bd09ded1f48fbf4"
server_key_iv: "e4c8f23e8789e50b9395037f187a7552"
server_key_version: "encryptionkey@0"
I20260929 18:37:20.016227 14478 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260929 18:37:20.018870 14493 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:37:20.020195 14478 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:37:20.020335 14478 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707036041078-13712-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707036041078-13712-0/minicluster-data/ts-1/wal
uuid: "e982a14ebab146c283af8dc996db9d72"
format_stamp: "Formatted at 2026-09-29 18:37:20 on dist-test-slave-j1xs"
server_key: "7cc2adcf6245e0855bd09ded1f48fbf4"
server_key_iv: "e4c8f23e8789e50b9395037f187a7552"
server_key_version: "encryptionkey@0"
I20260929 18:37:20.020452 14478 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707036041078-13712-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707036041078-13712-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707036041078-13712-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:37:20.037762 14478 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:37:20.041003 14478 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:37:20.041229 14478 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:37:20.041846 14478 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:37:20.042732 14478 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:37:20.042778 14478 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:37:20.042850 14478 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:37:20.042882 14478 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:37:20.051587 14478 rpc_server.cc:307] RPC server started. Bound to: 127.13.100.2:45001
I20260929 18:37:20.051618 14606 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.100.2:45001 every 8 connection(s)
I20260929 18:37:20.052495 14478 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707036041078-13712-0/minicluster-data/ts-1/data/info.pb
I20260929 18:37:20.060473 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 14478
I20260929 18:37:20.060572 13712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707036041078-13712-0/minicluster-data/ts-1/wal/instance
I20260929 18:37:20.060840 13712 external_mini_cluster.cc:1428] Setting key 56e887e5486fcaaf71fab7c73562d1de
Sep 29 18:37:20 dist-test-slave-j1xs krb5kdc[14237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707040, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.2@KRBTEST.COM for kudu/127.13.100.62@KRBTEST.COM
I20260929 18:37:20.065706 14321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:20.054304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.2:50963 (local address 127.13.100.62:45687)
0929 18:37:20.054478 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:20.054482 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:20.055295 (+ 813us) server_negotiation.cc:407] Connection header received
0929 18:37:20.056194 (+ 899us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:20.056197 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:20.056253 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:20.056355 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:20.057568 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:20.058169 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:20.059256 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:20.059471 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:20.059579 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:20.062133 (+ 2554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:20.062139 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:20.062143 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:37:20.062167 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:20.063755 (+ 1588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:20.064255 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:20.064260 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:20.064262 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:20.064340 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:20.064668 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:20.064671 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:20.064673 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:20.064842 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:20.064949 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:20.065497 (+ 548us) server_negotiation.cc:299] Negotiation successful
0929 18:37:20.065591 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 18:37:20.066890 14609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:20.054696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:45687 (local address 127.13.100.2:50963)
0929 18:37:20.055138 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:20.055190 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:20.055991 (+ 801us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:20.056511 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:20.056518 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:20.056967 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:37:20.057430 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:20.057442 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:20.058285 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:20.058289 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:20.059130 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:20.059139 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:20.059620 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:20.059623 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:20.059779 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:20.060374 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:37:20.060389 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:37:20.061944 (+ 1555us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:37:20.063878 (+ 1934us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:20.063884 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:20.063897 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:20.064143 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:20.064432 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:20.064437 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:20.064439 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:20.064568 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:20.064944 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:37:20.064950 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:37:20.065245 (+ 295us) client_negotiation.cc:769] Sending connection context
0929 18:37:20.065445 (+ 200us) client_negotiation.cc:240] Negotiation successful
0929 18:37:20.065784 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":116,"threads_started":1}
I20260929 18:37:20.068030 14607 heartbeater.cc:347] Connected to a master server at 127.13.100.62:45687
I20260929 18:37:20.068275 14607 heartbeater.cc:464] Registering TS with master...
I20260929 18:37:20.068740 14607 heartbeater.cc:511] Master 127.13.100.62:45687 requested a full tablet report, sending...
I20260929 18:37:20.069729 14282 ts_manager.cc:194] Registered new tserver with Master: e982a14ebab146c283af8dc996db9d72 (127.13.100.2:45001)
I20260929 18:37:20.070326 14282 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.100.2@KRBTEST.COM'} at 127.13.100.2:50963
WARNING: no policy specified for kudu/127.13.100.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.100.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.100.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.3.keytab.
Entry for principal kudu/127.13.100.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.3.keytab.
I20260929 18:37:20.089126 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707036041078-13712-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707036041078-13712-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707036041078-13712-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707036041078-13712-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.100.3:0
--local_ip_for_outbound_sockets=127.13.100.3
--webserver_interface=127.13.100.3
--webserver_port=0
--tserver_master_addrs=127.13.100.62:45687
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.3.keytab
--principal=kudu/127.13.100.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.100.20:44345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707036041078-13712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707036041078-13712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707036041078-13712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:37:20.200141 14612 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:37:20.200377 14612 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:37:20.200423 14612 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:37:20.204365 14612 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:37:20.204418 14612 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:37:20.204478 14612 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.100.3
I20260929 18:37:20.209681 14612 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.100.20:44345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707036041078-13712-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707036041078-13712-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.3.keytab
--principal=kudu/127.13.100.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.100.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707036041078-13712-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.100.3
--webserver_port=0
--tserver_master_addrs=127.13.100.62:45687
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14612
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.100.3
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707036041078-13712-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:37:20.210698 14612 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:37:20.211633 14612 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:37:20.221175 14620 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:37:20.221182 14617 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:37:20.221182 14618 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:37:20.221256 14612 server_base.cc:1034] running on GCE node
I20260929 18:37:20.221815 14612 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:37:20.222430 14612 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:37:20.223626 14612 hybrid_clock.cc:648] HybridClock initialized: now 1790707040223611 us; error 28 us; skew 500 ppm
Sep 29 18:37:20 dist-test-slave-j1xs krb5kdc[14237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707040, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:37:20.226516 14612 init.cc:377] Logged in from keytab as kudu/127.13.100.3@KRBTEST.COM (short username kudu)
I20260929 18:37:20.227303 14612 webserver.cc:460] Webserver started at http://127.13.100.3:39519/ using document root <none> and password file <none>
I20260929 18:37:20.227813 14612 fs_manager.cc:359] Metadata directory not provided
I20260929 18:37:20.227871 14612 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:37:20.228024 14612 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:37:20.230000 14612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707036041078-13712-0/minicluster-data/ts-2/data/instance:
uuid: "ec12801afede48d792819268acb6e1cb"
format_stamp: "Formatted at 2026-09-29 18:37:20 on dist-test-slave-j1xs"
server_key: "b489be671722c0a7935700c103799bb9"
server_key_iv: "94367c04f7b53c216a597bee346591ef"
server_key_version: "encryptionkey@0"
I20260929 18:37:20.230489 14612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707036041078-13712-0/minicluster-data/ts-2/wal/instance:
uuid: "ec12801afede48d792819268acb6e1cb"
format_stamp: "Formatted at 2026-09-29 18:37:20 on dist-test-slave-j1xs"
server_key: "b489be671722c0a7935700c103799bb9"
server_key_iv: "94367c04f7b53c216a597bee346591ef"
server_key_version: "encryptionkey@0"
I20260929 18:37:20.234006 14612 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20260929 18:37:20.236384 14627 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:37:20.237795 14612 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:37:20.237974 14612 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707036041078-13712-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707036041078-13712-0/minicluster-data/ts-2/wal
uuid: "ec12801afede48d792819268acb6e1cb"
format_stamp: "Formatted at 2026-09-29 18:37:20 on dist-test-slave-j1xs"
server_key: "b489be671722c0a7935700c103799bb9"
server_key_iv: "94367c04f7b53c216a597bee346591ef"
server_key_version: "encryptionkey@0"
I20260929 18:37:20.238106 14612 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707036041078-13712-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707036041078-13712-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707036041078-13712-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:37:20.260231 14612 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:37:20.263324 14612 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:37:20.263528 14612 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:37:20.264127 14612 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:37:20.264930 14612 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:37:20.265000 14612 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:37:20.265079 14612 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:37:20.265133 14612 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:37:20.273739 14612 rpc_server.cc:307] RPC server started. Bound to: 127.13.100.3:43209
I20260929 18:37:20.273766 14740 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.100.3:43209 every 8 connection(s)
I20260929 18:37:20.274616 14612 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707036041078-13712-0/minicluster-data/ts-2/data/info.pb
I20260929 18:37:20.275785 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 14612
I20260929 18:37:20.275880 13712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707036041078-13712-0/minicluster-data/ts-2/wal/instance
I20260929 18:37:20.276170 13712 external_mini_cluster.cc:1428] Setting key 9ea3944d3d08ea8db97d2aeb2953b193
Sep 29 18:37:20 dist-test-slave-j1xs krb5kdc[14237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707040, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.3@KRBTEST.COM for kudu/127.13.100.62@KRBTEST.COM
I20260929 18:37:20.287838 14321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:20.276451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.3:59351 (local address 127.13.100.62:45687)
0929 18:37:20.276606 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:20.276613 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:20.277669 (+ 1056us) server_negotiation.cc:407] Connection header received
0929 18:37:20.278839 (+ 1170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:20.278841 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:20.278883 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:20.278949 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:20.280005 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:20.280531 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:20.281617 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:20.281840 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:20.282007 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:20.284309 (+ 2302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:20.284312 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:20.284314 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:37:20.284337 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:20.285786 (+ 1449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:20.286320 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:20.286325 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:20.286327 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:20.286408 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:20.286747 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:20.286752 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:20.286754 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:20.286931 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:20.287042 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:20.287527 (+ 485us) server_negotiation.cc:299] Negotiation successful
0929 18:37:20.287653 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 18:37:20.288681 14743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:20.276914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:45687 (local address 127.13.100.3:59351)
0929 18:37:20.277505 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:20.277571 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:20.278665 (+ 1094us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:20.279071 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:20.279077 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:20.279418 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:37:20.279877 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:20.279889 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:20.280649 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:20.280651 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:20.281498 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:20.281506 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:20.282049 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:20.282051 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:20.282189 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:20.282709 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:37:20.282724 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:37:20.284139 (+ 1415us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:37:20.285932 (+ 1793us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:20.285937 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:20.285952 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:20.286216 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:20.286507 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:20.286510 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:20.286512 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:20.286651 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:20.287051 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:37:20.287060 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:37:20.287299 (+ 239us) client_negotiation.cc:769] Sending connection context
0929 18:37:20.287486 (+ 187us) client_negotiation.cc:240] Negotiation successful
0929 18:37:20.287811 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":191,"threads_started":1}
I20260929 18:37:20.289944 14741 heartbeater.cc:347] Connected to a master server at 127.13.100.62:45687
I20260929 18:37:20.290220 14741 heartbeater.cc:464] Registering TS with master...
I20260929 18:37:20.290721 14741 heartbeater.cc:511] Master 127.13.100.62:45687 requested a full tablet report, sending...
I20260929 18:37:20.291638 14282 ts_manager.cc:194] Registered new tserver with Master: ec12801afede48d792819268acb6e1cb (127.13.100.3:43209)
I20260929 18:37:20.292199 14282 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.100.3@KRBTEST.COM'} at 127.13.100.3:59351
I20260929 18:37:20.302379 13712 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 18:37:20.313565 14321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:20.306316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39856 (local address 127.13.100.62:45687)
0929 18:37:20.306443 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:20.306446 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:20.306560 (+ 114us) server_negotiation.cc:407] Connection header received
0929 18:37:20.306658 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:20.306661 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:20.306702 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:20.306775 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:20.307478 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:20.307998 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:20.308827 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:20.309082 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:20.309179 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:20.310649 (+ 1470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:20.310652 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:20.310654 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:37:20.310676 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:20.311927 (+ 1251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:20.312352 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:20.312355 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:20.312356 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:20.312444 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:20.312772 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:20.312775 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:20.312776 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:20.313000 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:20.313116 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:20.313355 (+ 239us) server_negotiation.cc:299] Negotiation successful
0929 18:37:20.313446 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 18:37:20.318396 14282 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39856:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:37:20.321341 14282 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:37:20.336540 14752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:20.330786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.3:43209 (local address 127.0.0.1:33738)
0929 18:37:20.331520 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:20.331546 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:20.331642 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:20.332330 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:20.332336 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:20.332357 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:37:20.332502 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:20.332508 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:20.333570 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:20.333575 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:20.334864 (+ 1289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:20.334874 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:20.335972 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:20.335979 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:20.336100 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:20.336139 (+ 39us) client_negotiation.cc:769] Sending connection context
0929 18:37:20.336254 (+ 115us) client_negotiation.cc:240] Negotiation successful
0929 18:37:20.336376 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":660,"thread_start_us":94,"threads_started":1}
I20260929 18:37:20.336577 14757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:20.331212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33738 (local address 127.13.100.3:43209)
0929 18:37:20.331801 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:20.331814 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:20.331839 (+ 25us) server_negotiation.cc:407] Connection header received
0929 18:37:20.331990 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:20.331997 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:20.332188 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:20.332335 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:37:20.332649 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:20.333437 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:20.335007 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:20.335835 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:20.335943 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:20.336087 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:20.336306 (+ 219us) server_negotiation.cc:299] Negotiation successful
0929 18:37:20.336404 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":485,"thread_start_us":223,"threads_started":1}
I20260929 18:37:20.336586 14756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:20.330561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.2:45001 (local address 127.0.0.1:35978)
0929 18:37:20.331958 (+ 1397us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:20.331976 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:20.332089 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:20.332603 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:20.332607 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:20.332628 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:37:20.332707 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:20.332713 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:20.334155 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:20.334158 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:20.335245 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:20.335254 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:20.336258 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:20.336262 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:20.336338 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:20.336358 (+ 20us) client_negotiation.cc:769] Sending connection context
0929 18:37:20.336422 (+ 64us) client_negotiation.cc:240] Negotiation successful
0929 18:37:20.336480 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1264,"mutex_wait_us":593,"thread_start_us":86,"threads_started":1}
I20260929 18:37:20.336936 14755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:20.330764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35978 (local address 127.13.100.2:45001)
0929 18:37:20.331441 (+ 677us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:20.331454 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:20.332189 (+ 735us) server_negotiation.cc:407] Connection header received
0929 18:37:20.332274 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:20.332279 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:20.332449 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:20.332614 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:37:20.332781 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:20.334007 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:20.335365 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:20.335944 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:20.336047 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:20.336196 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:20.336680 (+ 484us) server_negotiation.cc:299] Negotiation successful
0929 18:37:20.336777 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":590,"thread_start_us":113,"threads_started":1}
I20260929 18:37:20.336966 14754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:20.330561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.1:34283 (local address 127.0.0.1:56180)
0929 18:37:20.331403 (+ 842us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:20.331452 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:20.331584 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:20.332484 (+ 900us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:20.332487 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:20.332506 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:37:20.332642 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:20.332651 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:20.334102 (+ 1451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:20.334106 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:20.335325 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:20.335334 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:20.336519 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:20.336523 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:20.336632 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:20.336660 (+ 28us) client_negotiation.cc:769] Sending connection context
0929 18:37:20.336737 (+ 77us) client_negotiation.cc:240] Negotiation successful
0929 18:37:20.336808 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":653,"mutex_wait_us":31,"thread_start_us":86,"threads_started":1}
I20260929 18:37:20.337476 14753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:20.330720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56180 (local address 127.13.100.1:34283)
0929 18:37:20.331248 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:20.331263 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:20.331728 (+ 465us) server_negotiation.cc:407] Connection header received
0929 18:37:20.331811 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:20.331817 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:20.332176 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:20.332324 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:37:20.332774 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:20.333960 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:20.335452 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:20.336245 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:20.336345 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:20.336458 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:20.337262 (+ 804us) server_negotiation.cc:299] Negotiation successful
0929 18:37:20.337350 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":87,"threads_started":1}
I20260929 18:37:20.339947 14407 tablet_service.cc:1467] Processing CreateTablet for tablet 6c8ccebee39d4797ae8fb6f5a23d25e2 (DEFAULT_TABLE table=test-table [id=1c014164432241d6a5649f3679372555]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:37:20.340003 14541 tablet_service.cc:1467] Processing CreateTablet for tablet 6c8ccebee39d4797ae8fb6f5a23d25e2 (DEFAULT_TABLE table=test-table [id=1c014164432241d6a5649f3679372555]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:37:20.340349 14675 tablet_service.cc:1467] Processing CreateTablet for tablet 6c8ccebee39d4797ae8fb6f5a23d25e2 (DEFAULT_TABLE table=test-table [id=1c014164432241d6a5649f3679372555]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:37:20.341007 14407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c8ccebee39d4797ae8fb6f5a23d25e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:37:20.341019 14541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c8ccebee39d4797ae8fb6f5a23d25e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:37:20.341236 14675 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c8ccebee39d4797ae8fb6f5a23d25e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:37:20.347054 14758 tablet_bootstrap.cc:492] T 6c8ccebee39d4797ae8fb6f5a23d25e2 P 02a35256f86d41908c390a07804ef794: Bootstrap starting.
I20260929 18:37:20.348361 14760 tablet_bootstrap.cc:492] T 6c8ccebee39d4797ae8fb6f5a23d25e2 P ec12801afede48d792819268acb6e1cb: Bootstrap starting.
I20260929 18:37:20.350124 14758 tablet_bootstrap.cc:654] T 6c8ccebee39d4797ae8fb6f5a23d25e2 P 02a35256f86d41908c390a07804ef794: Neither blocks nor log segments found. Creating new log.
I20260929 18:37:20.350287 14760 tablet_bootstrap.cc:654] T 6c8ccebee39d4797ae8fb6f5a23d25e2 P ec12801afede48d792819268acb6e1cb: Neither blocks nor log segments found. Creating new log.
I20260929 18:37:20.350283 14759 tablet_bootstrap.cc:492] T 6c8ccebee39d4797ae8fb6f5a23d25e2 P e982a14ebab146c283af8dc996db9d72: Bootstrap starting.
I20260929 18:37:20.351873 14758 log.cc:824] T 6c8ccebee39d4797ae8fb6f5a23d25e2 P 02a35256f86d41908c390a07804ef794: Log is configured to *not* fsync() on all Append() calls
I20260929 18:37:20.351873 14760 log.cc:824] T 6c8ccebee39d4797ae8fb6f5a23d25e2 P ec12801afede48d792819268acb6e1cb: Log is configured to *not* fsync() on all Append() calls
I20260929 18:37:20.352264 14759 tablet_bootstrap.cc:654] T 6c8ccebee39d4797ae8fb6f5a23d25e2 P e982a14ebab146c283af8dc996db9d72: Neither blocks nor log segments found. Creating new log.
I20260929 18:37:20.352986 14759 log.cc:824] T 6c8ccebee39d4797ae8fb6f5a23d25e2 P e982a14ebab146c283af8dc996db9d72: Log is configured to *not* fsync() on all Append() calls
I20260929 18:37:20.353866 14760 tablet_bootstrap.cc:492] T 6c8ccebee39d4797ae8fb6f5a23d25e2 P ec12801afede48d792819268acb6e1cb: No bootstrap required, opened a new log
I20260929 18:37:20.354105 14758 tablet_bootstrap.cc:492] T 6c8ccebee39d4797ae8fb6f5a23d25e2 P 02a35256f86d41908c390a07804ef794: No bootstrap required, opened a new log
I20260929 18:37:20.354118 14760 ts_tablet_manager.cc:1397] T 6c8ccebee39d4797ae8fb6f5a23d25e2 P ec12801afede48d792819268acb6e1cb: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.001s
I20260929 18:37:20.354280 14758 ts_tablet_manager.cc:1397] T 6c8ccebee39d4797ae8fb6f5a23d25e2 P 02a35256f86d41908c390a07804ef794: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260929 18:37:20.354984 14759 tablet_bootstrap.cc:492] T 6c8ccebee39d4797ae8fb6f5a23d25e2 P e982a14ebab146c283af8dc996db9d72: No bootstrap required, opened a new log
I20260929 18:37:20.355188 14759 ts_tablet_manager.cc:1397] T 6c8ccebee39d4797ae8fb6f5a23d25e2 P e982a14ebab146c283af8dc996db9d72: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260929 18:37:20.357820 14758 raft_consensus.cc:348] T 6c8ccebee39d4797ae8fb6f5a23d25e2 P 02a35256f86d41908c390a07804ef794 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02a35256f86d41908c390a07804ef794" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 34283 } } peers { permanent_uuid: "e982a14ebab146c283af8dc996db9d72" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 45001 } } peers { permanent_uuid: "ec12801afede48d792819268acb6e1cb" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 43209 } }
I20260929 18:37:20.357841 14759 raft_consensus.cc:348] T 6c8ccebee39d4797ae8fb6f5a23d25e2 P e982a14ebab146c283af8dc996db9d72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02a35256f86d41908c390a07804ef794" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 34283 } } peers { permanent_uuid: "e982a14ebab146c283af8dc996db9d72" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 45001 } } peers { permanent_uuid: "ec12801afede48d792819268acb6e1cb" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 43209 } }
I20260929 18:37:20.357820 14760 raft_consensus.cc:348] T 6c8ccebee39d4797ae8fb6f5a23d25e2 P ec12801afede48d792819268acb6e1cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02a35256f86d41908c390a07804ef794" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 34283 } } peers { permanent_uuid: "e982a14ebab146c283af8dc996db9d72" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 45001 } } peers { permanent_uuid: "ec12801afede48d792819268acb6e1cb" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 43209 } }
I20260929 18:37:20.358101 14758 raft_consensus.cc:374] T 6c8ccebee39d4797ae8fb6f5a23d25e2 P 02a35256f86d41908c390a07804ef794 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:37:20.358119 14760 raft_consensus.cc:374] T 6c8ccebee39d4797ae8fb6f5a23d25e2 P ec12801afede48d792819268acb6e1cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:37:20.358124 14759 raft_consensus.cc:374] T 6c8ccebee39d4797ae8fb6f5a23d25e2 P e982a14ebab146c283af8dc996db9d72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:37:20.358174 14759 raft_consensus.cc:729] T 6c8ccebee39d4797ae8fb6f5a23d25e2 P e982a14ebab146c283af8dc996db9d72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e982a14ebab146c283af8dc996db9d72, State: Initialized, Role: FOLLOWER
I20260929 18:37:20.358170 14760 raft_consensus.cc:729] T 6c8ccebee39d4797ae8fb6f5a23d25e2 P ec12801afede48d792819268acb6e1cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec12801afede48d792819268acb6e1cb, State: Initialized, Role: FOLLOWER
I20260929 18:37:20.358163 14758 raft_consensus.cc:729] T 6c8ccebee39d4797ae8fb6f5a23d25e2 P 02a35256f86d41908c390a07804ef794 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02a35256f86d41908c390a07804ef794, State: Initialized, Role: FOLLOWER
I20260929 18:37:20.358593 14759 consensus_queue.cc:260] T 6c8ccebee39d4797ae8fb6f5a23d25e2 P e982a14ebab146c283af8dc996db9d72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02a35256f86d41908c390a07804ef794" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 34283 } } peers { permanent_uuid: "e982a14ebab146c283af8dc996db9d72" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 45001 } } peers { permanent_uuid: "ec12801afede48d792819268acb6e1cb" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 43209 } }
I20260929 18:37:20.358610 14758 consensus_queue.cc:260] T 6c8ccebee39d4797ae8fb6f5a23d25e2 P 02a35256f86d41908c390a07804ef794 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02a35256f86d41908c390a07804ef794" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 34283 } } peers { permanent_uuid: "e982a14ebab146c283af8dc996db9d72" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 45001 } } peers { permanent_uuid: "ec12801afede48d792819268acb6e1cb" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 43209 } }
I20260929 18:37:20.358603 14760 consensus_queue.cc:260] T 6c8ccebee39d4797ae8fb6f5a23d25e2 P ec12801afede48d792819268acb6e1cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02a35256f86d41908c390a07804ef794" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 34283 } } peers { permanent_uuid: "e982a14ebab146c283af8dc996db9d72" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 45001 } } peers { permanent_uuid: "ec12801afede48d792819268acb6e1cb" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 43209 } }
I20260929 18:37:20.359179 14473 heartbeater.cc:503] Master 127.13.100.62:45687 was elected leader, sending a full tablet report...
I20260929 18:37:20.359529 14760 ts_tablet_manager.cc:1428] T 6c8ccebee39d4797ae8fb6f5a23d25e2 P ec12801afede48d792819268acb6e1cb: Time spent starting tablet: real 0.005s user 0.003s sys 0.002s
I20260929 18:37:20.359529 14758 ts_tablet_manager.cc:1428] T 6c8ccebee39d4797ae8fb6f5a23d25e2 P 02a35256f86d41908c390a07804ef794: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260929 18:37:20.359858 14741 heartbeater.cc:503] Master 127.13.100.62:45687 was elected leader, sending a full tablet report...
I20260929 18:37:20.360381 14759 ts_tablet_manager.cc:1428] T 6c8ccebee39d4797ae8fb6f5a23d25e2 P e982a14ebab146c283af8dc996db9d72: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260929 18:37:20.360771 14607 heartbeater.cc:503] Master 127.13.100.62:45687 was elected leader, sending a full tablet report...
I20260929 18:37:20.460251 14766 raft_consensus.cc:482] T 6c8ccebee39d4797ae8fb6f5a23d25e2 P ec12801afede48d792819268acb6e1cb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:37:20.460495 14766 raft_consensus.cc:504] T 6c8ccebee39d4797ae8fb6f5a23d25e2 P ec12801afede48d792819268acb6e1cb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02a35256f86d41908c390a07804ef794" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 34283 } } peers { permanent_uuid: "e982a14ebab146c283af8dc996db9d72" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 45001 } } peers { permanent_uuid: "ec12801afede48d792819268acb6e1cb" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 43209 } }
I20260929 18:37:20.461687 14766 leader_election.cc:288] T 6c8ccebee39d4797ae8fb6f5a23d25e2 P ec12801afede48d792819268acb6e1cb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 02a35256f86d41908c390a07804ef794 (127.13.100.1:34283), e982a14ebab146c283af8dc996db9d72 (127.13.100.2:45001)
I20260929 18:37:20.464650 14764 raft_consensus.cc:482] T 6c8ccebee39d4797ae8fb6f5a23d25e2 P 02a35256f86d41908c390a07804ef794 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:37:20.464820 14764 raft_consensus.cc:504] T 6c8ccebee39d4797ae8fb6f5a23d25e2 P 02a35256f86d41908c390a07804ef794 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02a35256f86d41908c390a07804ef794" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 34283 } } peers { permanent_uuid: "e982a14ebab146c283af8dc996db9d72" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 45001 } } peers { permanent_uuid: "ec12801afede48d792819268acb6e1cb" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 43209 } }
I20260929 18:37:20.466225 14743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:20.461874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.1:34283 (local address 127.13.100.3:49933)
0929 18:37:20.462082 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:20.462098 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:20.462188 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:20.462516 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:20.462522 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:20.462539 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:37:20.462648 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:20.462653 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:20.463703 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:20.463705 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:20.465061 (+ 1356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:20.465071 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:20.465889 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:20.465893 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:20.466014 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:20.466030 (+ 16us) client_negotiation.cc:769] Sending connection context
0929 18:37:20.466090 (+ 60us) client_negotiation.cc:240] Negotiation successful
0929 18:37:20.466145 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"mutex_wait_us":29}
I20260929 18:37:20.466274 14755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:20.462086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.3:59255 (local address 127.13.100.2:45001)
0929 18:37:20.462220 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:20.462225 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:20.462447 (+ 222us) server_negotiation.cc:407] Connection header received
0929 18:37:20.462565 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:20.462569 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:20.462627 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:20.462687 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:37:20.462946 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:20.463673 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:20.465221 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:20.465642 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:20.465772 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:20.465822 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:20.466012 (+ 190us) server_negotiation.cc:299] Negotiation successful
0929 18:37:20.466125 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 18:37:20.466238 14767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:20.461968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.2:45001 (local address 127.13.100.3:59255)
0929 18:37:20.462356 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:20.462367 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:20.462439 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:20.462725 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:20.462729 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:20.462748 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:37:20.462848 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:20.462852 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:20.463773 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:20.463776 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:20.465084 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:20.465089 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:20.465764 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:20.465767 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:20.465869 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:20.465917 (+ 48us) client_negotiation.cc:769] Sending connection context
0929 18:37:20.465990 (+ 73us) client_negotiation.cc:240] Negotiation successful
0929 18:37:20.466068 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"mutex_wait_us":76,"thread_start_us":115,"threads_started":1}
I20260929 18:37:20.466512 14753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:20.462027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.3:49933 (local address 127.13.100.1:34283)
0929 18:37:20.462151 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:20.462156 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:20.462169 (+ 13us) server_negotiation.cc:407] Connection header received
0929 18:37:20.462329 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:20.462331 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:20.462376 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:20.462479 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:37:20.462789 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:20.463586 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:20.465187 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:20.465725 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:20.465838 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:20.465909 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:20.466297 (+ 388us) server_negotiation.cc:299] Negotiation successful
0929 18:37:20.466381 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 18:37:20.467187 14427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8ccebee39d4797ae8fb6f5a23d25e2" candidate_uuid: "ec12801afede48d792819268acb6e1cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02a35256f86d41908c390a07804ef794" is_pre_election: true
I20260929 18:37:20.467432 14427 raft_consensus.cc:2454] T 6c8ccebee39d4797ae8fb6f5a23d25e2 P 02a35256f86d41908c390a07804ef794 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec12801afede48d792819268acb6e1cb in term 0.
I20260929 18:37:20.467609 14561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8ccebee39d4797ae8fb6f5a23d25e2" candidate_uuid: "ec12801afede48d792819268acb6e1cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e982a14ebab146c283af8dc996db9d72" is_pre_election: true
I20260929 18:37:20.467904 14561 raft_consensus.cc:2454] T 6c8ccebee39d4797ae8fb6f5a23d25e2 P e982a14ebab146c283af8dc996db9d72 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec12801afede48d792819268acb6e1cb in term 0.
I20260929 18:37:20.467918 14628 leader_election.cc:302] T 6c8ccebee39d4797ae8fb6f5a23d25e2 P ec12801afede48d792819268acb6e1cb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02a35256f86d41908c390a07804ef794, ec12801afede48d792819268acb6e1cb; no voters:
I20260929 18:37:20.468199 14766 raft_consensus.cc:2781] T 6c8ccebee39d4797ae8fb6f5a23d25e2 P ec12801afede48d792819268acb6e1cb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:37:20.468276 14766 raft_consensus.cc:482] T 6c8ccebee39d4797ae8fb6f5a23d25e2 P ec12801afede48d792819268acb6e1cb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:37:20.468319 14766 raft_consensus.cc:3037] T 6c8ccebee39d4797ae8fb6f5a23d25e2 P ec12801afede48d792819268acb6e1cb [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:37:20.467204 14764 leader_election.cc:288] T 6c8ccebee39d4797ae8fb6f5a23d25e2 P 02a35256f86d41908c390a07804ef794 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e982a14ebab146c283af8dc996db9d72 (127.13.100.2:45001), ec12801afede48d792819268acb6e1cb (127.13.100.3:43209)
I20260929 18:37:20.469686 14766 raft_consensus.cc:504] T 6c8ccebee39d4797ae8fb6f5a23d25e2 P ec12801afede48d792819268acb6e1cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02a35256f86d41908c390a07804ef794" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 34283 } } peers { permanent_uuid: "e982a14ebab146c283af8dc996db9d72" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 45001 } } peers { permanent_uuid: "ec12801afede48d792819268acb6e1cb" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 43209 } }
I20260929 18:37:20.470047 14766 leader_election.cc:288] T 6c8ccebee39d4797ae8fb6f5a23d25e2 P ec12801afede48d792819268acb6e1cb [CANDIDATE]: Term 1 election: Requested vote from peers 02a35256f86d41908c390a07804ef794 (127.13.100.1:34283), e982a14ebab146c283af8dc996db9d72 (127.13.100.2:45001)
I20260929 18:37:20.470412 14427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8ccebee39d4797ae8fb6f5a23d25e2" candidate_uuid: "ec12801afede48d792819268acb6e1cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02a35256f86d41908c390a07804ef794"
I20260929 18:37:20.470430 14561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8ccebee39d4797ae8fb6f5a23d25e2" candidate_uuid: "ec12801afede48d792819268acb6e1cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e982a14ebab146c283af8dc996db9d72"
I20260929 18:37:20.470518 14427 raft_consensus.cc:3037] T 6c8ccebee39d4797ae8fb6f5a23d25e2 P 02a35256f86d41908c390a07804ef794 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:37:20.470518 14561 raft_consensus.cc:3037] T 6c8ccebee39d4797ae8fb6f5a23d25e2 P e982a14ebab146c283af8dc996db9d72 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:37:20.471438 14757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:20.467146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.1:54303 (local address 127.13.100.3:43209)
0929 18:37:20.467277 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:20.467281 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:20.467292 (+ 11us) server_negotiation.cc:407] Connection header received
0929 18:37:20.467324 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:20.467327 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:20.467362 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:20.467428 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:37:20.467841 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:20.468493 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:20.470328 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:20.470870 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:20.471000 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:20.471049 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:20.471253 (+ 204us) server_negotiation.cc:299] Negotiation successful
0929 18:37:20.471330 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 18:37:20.471534 14755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:20.466486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.1:45303 (local address 127.13.100.2:45001)
0929 18:37:20.466629 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:20.466633 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:20.466851 (+ 218us) server_negotiation.cc:407] Connection header received
0929 18:37:20.467050 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:20.467056 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:20.467108 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:20.467153 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:37:20.467541 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:20.468271 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:20.469978 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:20.470559 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:20.470662 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:20.470702 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:20.471366 (+ 664us) server_negotiation.cc:299] Negotiation successful
0929 18:37:20.471424 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260929 18:37:20.471711 14561 raft_consensus.cc:2454] T 6c8ccebee39d4797ae8fb6f5a23d25e2 P e982a14ebab146c283af8dc996db9d72 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec12801afede48d792819268acb6e1cb in term 1.
I20260929 18:37:20.471711 14427 raft_consensus.cc:2454] T 6c8ccebee39d4797ae8fb6f5a23d25e2 P 02a35256f86d41908c390a07804ef794 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec12801afede48d792819268acb6e1cb in term 1.
I20260929 18:37:20.471865 14768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:20.466117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.2:45001 (local address 127.13.100.1:45303)
0929 18:37:20.466754 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:20.466782 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:20.466916 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:20.467295 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:20.467299 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:20.467319 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:37:20.467426 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:20.467432 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:20.468393 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:20.468396 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:20.469779 (+ 1383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:20.469813 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:20.471100 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:20.471103 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:20.471243 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:20.471264 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 18:37:20.471643 (+ 379us) client_negotiation.cc:240] Negotiation successful
0929 18:37:20.471714 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"mutex_wait_us":77,"thread_start_us":118,"threads_started":1}
I20260929 18:37:20.472123 14769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:20.466627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.3:43209 (local address 127.13.100.1:54303)
0929 18:37:20.466934 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:20.466948 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:20.467026 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:20.467562 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:20.467564 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:20.467580 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:37:20.467673 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:20.467678 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:20.469035 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:20.469038 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:20.470185 (+ 1147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:20.470199 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:20.471000 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:20.471004 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:20.471099 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:20.471140 (+ 41us) client_negotiation.cc:769] Sending connection context
0929 18:37:20.471753 (+ 613us) client_negotiation.cc:240] Negotiation successful
0929 18:37:20.471856 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"mutex_wait_us":5,"thread_start_us":87,"threads_started":1}
I20260929 18:37:20.472082 14631 leader_election.cc:302] T 6c8ccebee39d4797ae8fb6f5a23d25e2 P ec12801afede48d792819268acb6e1cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e982a14ebab146c283af8dc996db9d72, ec12801afede48d792819268acb6e1cb; no voters:
I20260929 18:37:20.472352 14766 raft_consensus.cc:2781] T 6c8ccebee39d4797ae8fb6f5a23d25e2 P ec12801afede48d792819268acb6e1cb [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:37:20.472468 14561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8ccebee39d4797ae8fb6f5a23d25e2" candidate_uuid: "02a35256f86d41908c390a07804ef794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e982a14ebab146c283af8dc996db9d72" is_pre_election: true
I20260929 18:37:20.472563 14766 raft_consensus.cc:686] T 6c8ccebee39d4797ae8fb6f5a23d25e2 P ec12801afede48d792819268acb6e1cb [term 1 LEADER]: Becoming Leader. State: Replica: ec12801afede48d792819268acb6e1cb, State: Running, Role: LEADER
I20260929 18:37:20.472637 14561 raft_consensus.cc:2379] T 6c8ccebee39d4797ae8fb6f5a23d25e2 P e982a14ebab146c283af8dc996db9d72 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 02a35256f86d41908c390a07804ef794 in current term 1: Already voted for candidate ec12801afede48d792819268acb6e1cb in this term.
I20260929 18:37:20.472819 14695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8ccebee39d4797ae8fb6f5a23d25e2" candidate_uuid: "02a35256f86d41908c390a07804ef794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec12801afede48d792819268acb6e1cb" is_pre_election: true
I20260929 18:37:20.472883 14766 consensus_queue.cc:237] T 6c8ccebee39d4797ae8fb6f5a23d25e2 P ec12801afede48d792819268acb6e1cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02a35256f86d41908c390a07804ef794" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 34283 } } peers { permanent_uuid: "e982a14ebab146c283af8dc996db9d72" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 45001 } } peers { permanent_uuid: "ec12801afede48d792819268acb6e1cb" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 43209 } }
I20260929 18:37:20.473330 14362 leader_election.cc:302] T 6c8ccebee39d4797ae8fb6f5a23d25e2 P 02a35256f86d41908c390a07804ef794 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02a35256f86d41908c390a07804ef794; no voters: e982a14ebab146c283af8dc996db9d72, ec12801afede48d792819268acb6e1cb
I20260929 18:37:20.473593 14764 raft_consensus.cc:2726] T 6c8ccebee39d4797ae8fb6f5a23d25e2 P 02a35256f86d41908c390a07804ef794 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 18:37:20.475927 14282 catalog_manager.cc:5495] T 6c8ccebee39d4797ae8fb6f5a23d25e2 P ec12801afede48d792819268acb6e1cb reported cstate change: term changed from 0 to 1, leader changed from <none> to ec12801afede48d792819268acb6e1cb (127.13.100.3). New cstate: current_term: 1 leader_uuid: "ec12801afede48d792819268acb6e1cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02a35256f86d41908c390a07804ef794" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 34283 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e982a14ebab146c283af8dc996db9d72" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 45001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec12801afede48d792819268acb6e1cb" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 43209 } health_report { overall_health: HEALTHY } } }
Sep 29 18:37:20 dist-test-slave-j1xs krb5kdc[14237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707039, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.100.1@KRBTEST.COM
I20260929 18:37:20.491392 14753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:20.483062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56194 (local address 127.13.100.1:34283)
0929 18:37:20.483197 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:20.483203 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:20.483421 (+ 218us) server_negotiation.cc:407] Connection header received
0929 18:37:20.483559 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:20.483562 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:20.483610 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:20.483694 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:20.484352 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:20.484848 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:20.485678 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:20.485953 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:20.486057 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:20.488407 (+ 2350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:20.488416 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:20.488428 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:37:20.488450 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:20.489863 (+ 1413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:20.490300 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:20.490304 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:20.490306 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:20.490392 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:20.490626 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:20.490628 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:20.490629 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:20.490844 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:20.490987 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:20.491194 (+ 207us) server_negotiation.cc:299] Negotiation successful
0929 18:37:20.491283 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
Sep 29 18:37:20 dist-test-slave-j1xs krb5kdc[14237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707040, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 18:37:20 dist-test-slave-j1xs krb5kdc[14237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707040, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.13.100.1@KRBTEST.COM
I20260929 18:37:20.517272 14753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:20.509300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56200 (local address 127.13.100.1:34283)
0929 18:37:20.509447 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:20.509453 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:20.509623 (+ 170us) server_negotiation.cc:407] Connection header received
0929 18:37:20.509799 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:20.509802 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:20.509840 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:20.509949 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:20.510623 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:20.511126 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:20.511950 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:20.512172 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:20.512280 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:20.514579 (+ 2299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:20.514582 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:20.514584 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:37:20.514607 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:20.515806 (+ 1199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:20.516159 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:20.516162 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:20.516163 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:20.516229 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:20.516519 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:20.516525 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:20.516527 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:20.516686 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:20.516786 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:20.517021 (+ 235us) server_negotiation.cc:299] Negotiation successful
0929 18:37:20.517153 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
W20260929 18:37:20.525686 14742 tablet.cc:2369] T 6c8ccebee39d4797ae8fb6f5a23d25e2 P ec12801afede48d792819268acb6e1cb: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 29 18:37:20 dist-test-slave-j1xs krb5kdc[14237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707040, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 18:37:20 dist-test-slave-j1xs krb5kdc[14237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707040, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.100.1@KRBTEST.COM
I20260929 18:37:20.541536 14753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:20.533752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56210 (local address 127.13.100.1:34283)
0929 18:37:20.533951 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:20.533955 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:20.534064 (+ 109us) server_negotiation.cc:407] Connection header received
0929 18:37:20.534157 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:20.534159 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:20.534198 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:20.534270 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:20.534885 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:20.535371 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:20.536131 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:20.536329 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:20.536405 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:20.538693 (+ 2288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:20.538696 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:20.538698 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:37:20.538718 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:20.540049 (+ 1331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:20.540449 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:20.540452 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:20.540453 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:20.540523 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:20.540745 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:20.540747 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:20.540748 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:20.540890 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:20.541003 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:20.541199 (+ 196us) server_negotiation.cc:299] Negotiation successful
0929 18:37:20.541286 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
W20260929 18:37:20.543025 14387 tablet_service.cc:2908] Rejecting scan request for tablet 6c8ccebee39d4797ae8fb6f5a23d25e2: Uninitialized: clean time has not yet been initialized
I20260929 18:37:20.545317 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 14344
I20260929 18:37:20.551165 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 14478
I20260929 18:37:20.557035 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 14612
I20260929 18:37:20.563354 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 14250
2026-09-29T18:37:20Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2360 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707036041078-13712-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:37:20 dist-test-slave-j1xs krb5kdc[14784](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:37:20 dist-test-slave-j1xs krb5kdc[14784](info): set up 2 sockets
Sep 29 18:37:20 dist-test-slave-j1xs krb5kdc[14784](info): commencing operation
krb5kdc: starting...
W20260929 18:37:21.599228 13712 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707036041078-13712-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707036041078-13712-0/krb5kdc/test-user.keytab.
Sep 29 18:37:21 dist-test-slave-j1xs krb5kdc[14784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707041, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:37:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:37:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.100.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.100.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.100.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.62.keytab.
Entry for principal kudu/127.13.100.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.62.keytab.
I20260929 18:37:21.717320 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707036041078-13712-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707036041078-13712-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707036041078-13712-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.100.62:34301
--webserver_interface=127.13.100.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.62.keytab
--principal=kudu/127.13.100.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.100.20:38841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.100.62:34301
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707036041078-13712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707036041078-13712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707036041078-13712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:37:21.823856 14797 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:37:21.824141 14797 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:37:21.824219 14797 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:37:21.828132 14797 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:37:21.828212 14797 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 18:37:21.828244 14797 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:37:21.828285 14797 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:37:21.828326 14797 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:37:21.828361 14797 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:37:21.833436 14797 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.100.20:38841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707036041078-13712-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707036041078-13712-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.100.62:34301
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.62.keytab
--principal=kudu/127.13.100.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.100.62:34301
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.100.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14797
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707036041078-13712-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:37:21.834442 14797 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:37:21.835426 14797 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:37:21.843647 14805 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:37:21.843664 14803 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:37:21.843715 14797 server_base.cc:1034] running on GCE node
W20260929 18:37:21.843664 14802 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:37:21.844487 14797 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:37:21.845577 14797 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:37:21.846771 14797 hybrid_clock.cc:648] HybridClock initialized: now 1790707041846739 us; error 42 us; skew 500 ppm
Sep 29 18:37:21 dist-test-slave-j1xs krb5kdc[14784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707041, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:37:21.854612 14797 init.cc:377] Logged in from keytab as kudu/127.13.100.62@KRBTEST.COM (short username kudu)
I20260929 18:37:21.855468 14797 webserver.cc:460] Webserver started at http://127.13.100.62:42829/ using document root <none> and password file <none>
I20260929 18:37:21.855963 14797 fs_manager.cc:359] Metadata directory not provided
I20260929 18:37:21.856027 14797 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:37:21.856184 14797 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:37:21.858184 14797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707036041078-13712-0/minicluster-data/master-0/data/instance:
uuid: "8da0f949c56c4288b898ad0637fee260"
format_stamp: "Formatted at 2026-09-29 18:37:21 on dist-test-slave-j1xs"
server_key: "f6ef894b3aeb4e8cfc63f95a4fbfab18"
server_key_iv: "ab4d7e8e01258f6bea80510ef218d52b"
server_key_version: "encryptionkey@0"
I20260929 18:37:21.858666 14797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707036041078-13712-0/minicluster-data/master-0/wal/instance:
uuid: "8da0f949c56c4288b898ad0637fee260"
format_stamp: "Formatted at 2026-09-29 18:37:21 on dist-test-slave-j1xs"
server_key: "f6ef894b3aeb4e8cfc63f95a4fbfab18"
server_key_iv: "ab4d7e8e01258f6bea80510ef218d52b"
server_key_version: "encryptionkey@0"
I20260929 18:37:21.862147 14797 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260929 18:37:21.864375 14812 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:37:21.865525 14797 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:37:21.865661 14797 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707036041078-13712-0/minicluster-data/master-0/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707036041078-13712-0/minicluster-data/master-0/wal
uuid: "8da0f949c56c4288b898ad0637fee260"
format_stamp: "Formatted at 2026-09-29 18:37:21 on dist-test-slave-j1xs"
server_key: "f6ef894b3aeb4e8cfc63f95a4fbfab18"
server_key_iv: "ab4d7e8e01258f6bea80510ef218d52b"
server_key_version: "encryptionkey@0"
I20260929 18:37:21.865765 14797 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707036041078-13712-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707036041078-13712-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707036041078-13712-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:37:21.884902 14797 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:37:21.887781 14797 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:37:21.887972 14797 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:37:21.895941 14797 rpc_server.cc:307] RPC server started. Bound to: 127.13.100.62:34301
I20260929 18:37:21.895959 14874 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.100.62:34301 every 8 connection(s)
I20260929 18:37:21.896847 14797 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
I20260929 18:37:21.899526 14875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:37:21.903301 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 14797
I20260929 18:37:21.903395 13712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707036041078-13712-0/minicluster-data/master-0/wal/instance
I20260929 18:37:21.903666 13712 external_mini_cluster.cc:1428] Setting key dcc5a36110c164a6d649d37065958132
I20260929 18:37:21.904951 14875 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8da0f949c56c4288b898ad0637fee260: Bootstrap starting.
I20260929 18:37:21.907603 14875 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8da0f949c56c4288b898ad0637fee260: Neither blocks nor log segments found. Creating new log.
I20260929 18:37:21.908501 14875 log.cc:824] T 00000000000000000000000000000000 P 8da0f949c56c4288b898ad0637fee260: Log is configured to *not* fsync() on all Append() calls
I20260929 18:37:21.910609 14875 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8da0f949c56c4288b898ad0637fee260: No bootstrap required, opened a new log
Sep 29 18:37:21 dist-test-slave-j1xs krb5kdc[14784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707041, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.100.62@KRBTEST.COM
I20260929 18:37:21.915192 14875 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8da0f949c56c4288b898ad0637fee260 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8da0f949c56c4288b898ad0637fee260" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 34301 } }
I20260929 18:37:21.915460 14875 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8da0f949c56c4288b898ad0637fee260 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:37:21.915524 14875 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8da0f949c56c4288b898ad0637fee260 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8da0f949c56c4288b898ad0637fee260, State: Initialized, Role: FOLLOWER
I20260929 18:37:21.916014 14875 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8da0f949c56c4288b898ad0637fee260 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8da0f949c56c4288b898ad0637fee260" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 34301 } }
I20260929 18:37:21.916142 14875 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8da0f949c56c4288b898ad0637fee260 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:37:21.916226 14875 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8da0f949c56c4288b898ad0637fee260 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:37:21.916328 14875 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8da0f949c56c4288b898ad0637fee260 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:37:21.917205 14875 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8da0f949c56c4288b898ad0637fee260 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8da0f949c56c4288b898ad0637fee260" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 34301 } }
I20260929 18:37:21.917551 14875 leader_election.cc:302] T 00000000000000000000000000000000 P 8da0f949c56c4288b898ad0637fee260 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8da0f949c56c4288b898ad0637fee260; no voters:
I20260929 18:37:21.917972 14875 leader_election.cc:288] T 00000000000000000000000000000000 P 8da0f949c56c4288b898ad0637fee260 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:37:21.918051 14880 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8da0f949c56c4288b898ad0637fee260 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:37:21.918264 14880 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8da0f949c56c4288b898ad0637fee260 [term 1 LEADER]: Becoming Leader. State: Replica: 8da0f949c56c4288b898ad0637fee260, State: Running, Role: LEADER
I20260929 18:37:21.918560 14880 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8da0f949c56c4288b898ad0637fee260 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8da0f949c56c4288b898ad0637fee260" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 34301 } }
I20260929 18:37:21.918999 14878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:21.905027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48594 (local address 127.13.100.62:34301)
0929 18:37:21.905469 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:21.905487 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:21.905528 (+ 41us) server_negotiation.cc:407] Connection header received
0929 18:37:21.906389 (+ 861us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:21.906413 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:21.906802 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:21.907155 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:21.907901 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:21.909218 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:21.910168 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:21.910810 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:21.910978 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:21.913709 (+ 2731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:21.913716 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:21.913732 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:37:21.913762 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:21.915743 (+ 1981us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:21.916317 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:21.916324 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:21.916330 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:21.916479 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:21.916781 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:21.916784 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:21.916786 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:21.917171 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:21.917375 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:21.917675 (+ 300us) server_negotiation.cc:299] Negotiation successful
0929 18:37:21.918024 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":114,"threads_started":1}
I20260929 18:37:21.919286 14875 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8da0f949c56c4288b898ad0637fee260 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:37:21.919760 14880 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8da0f949c56c4288b898ad0637fee260 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8da0f949c56c4288b898ad0637fee260. Latest consensus state: current_term: 1 leader_uuid: "8da0f949c56c4288b898ad0637fee260" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8da0f949c56c4288b898ad0637fee260" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 34301 } } }
I20260929 18:37:21.919863 14880 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8da0f949c56c4288b898ad0637fee260 [sys.catalog]: This master's current role is: LEADER
I20260929 18:37:21.920199 14881 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8da0f949c56c4288b898ad0637fee260 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8da0f949c56c4288b898ad0637fee260" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8da0f949c56c4288b898ad0637fee260" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 34301 } } }
I20260929 18:37:21.920305 14881 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8da0f949c56c4288b898ad0637fee260 [sys.catalog]: This master's current role is: LEADER
I20260929 18:37:21.921047 14884 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:37:21.924104 14884 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:37:21.930177 14884 catalog_manager.cc:1347] Generated new cluster ID: 16d3caf4f0664706aa9aecdd625a3a21
I20260929 18:37:21.930240 14884 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:37:21.942777 14884 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:37:21.943789 14884 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:37:21.952548 14884 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8da0f949c56c4288b898ad0637fee260: Generated new TSK 0
I20260929 18:37:21.953249 14884 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.100.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.100.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.100.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.1.keytab.
Entry for principal kudu/127.13.100.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.1.keytab.
I20260929 18:37:21.988327 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707036041078-13712-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707036041078-13712-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707036041078-13712-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707036041078-13712-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.100.1:0
--local_ip_for_outbound_sockets=127.13.100.1
--webserver_interface=127.13.100.1
--webserver_port=0
--tserver_master_addrs=127.13.100.62:34301
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.1.keytab
--principal=kudu/127.13.100.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.100.20:38841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707036041078-13712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707036041078-13712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707036041078-13712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:37:22.105064 14901 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:37:22.105332 14901 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:37:22.105522 14901 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:37:22.109645 14901 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:37:22.109735 14901 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:37:22.109802 14901 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 18:37:22.109915 14901 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.100.1
I20260929 18:37:22.115654 14901 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.100.20:38841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707036041078-13712-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707036041078-13712-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.1.keytab
--principal=kudu/127.13.100.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.100.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707036041078-13712-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.100.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.13.100.62:34301
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14901
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.100.1
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707036041078-13712-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:37:22.116709 14901 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:37:22.117715 14901 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 18:37:22.127674 14901 server_base.cc:1034] running on GCE node
W20260929 18:37:22.127563 14907 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:37:22.127599 14906 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:37:22.127568 14909 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:37:22.128211 14901 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:37:22.128875 14901 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:37:22.130112 14901 hybrid_clock.cc:648] HybridClock initialized: now 1790707042130084 us; error 42 us; skew 500 ppm
Sep 29 18:37:22 dist-test-slave-j1xs krb5kdc[14784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707042, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:37:22.132822 14901 init.cc:377] Logged in from keytab as kudu/127.13.100.1@KRBTEST.COM (short username kudu)
I20260929 18:37:22.133625 14901 webserver.cc:460] Webserver started at http://127.13.100.1:42973/ using document root <none> and password file <none>
I20260929 18:37:22.134202 14901 fs_manager.cc:359] Metadata directory not provided
I20260929 18:37:22.134274 14901 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:37:22.134435 14901 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:37:22.136380 14901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707036041078-13712-0/minicluster-data/ts-0/data/instance:
uuid: "fdfd0eb5ab40468594d7cb793150beb8"
format_stamp: "Formatted at 2026-09-29 18:37:22 on dist-test-slave-j1xs"
server_key: "4bce8a8460ff45fbb94a1ad6fe5fe2bc"
server_key_iv: "d586f101a718f264e7b34460a54c413f"
server_key_version: "encryptionkey@0"
I20260929 18:37:22.136874 14901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707036041078-13712-0/minicluster-data/ts-0/wal/instance:
uuid: "fdfd0eb5ab40468594d7cb793150beb8"
format_stamp: "Formatted at 2026-09-29 18:37:22 on dist-test-slave-j1xs"
server_key: "4bce8a8460ff45fbb94a1ad6fe5fe2bc"
server_key_iv: "d586f101a718f264e7b34460a54c413f"
server_key_version: "encryptionkey@0"
I20260929 18:37:22.140568 14901 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260929 18:37:22.142885 14916 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:37:22.144150 14901 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:37:22.144279 14901 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707036041078-13712-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707036041078-13712-0/minicluster-data/ts-0/wal
uuid: "fdfd0eb5ab40468594d7cb793150beb8"
format_stamp: "Formatted at 2026-09-29 18:37:22 on dist-test-slave-j1xs"
server_key: "4bce8a8460ff45fbb94a1ad6fe5fe2bc"
server_key_iv: "d586f101a718f264e7b34460a54c413f"
server_key_version: "encryptionkey@0"
I20260929 18:37:22.144382 14901 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707036041078-13712-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707036041078-13712-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707036041078-13712-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:37:22.153797 14901 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:37:22.156589 14901 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:37:22.156786 14901 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:37:22.158200 14901 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:37:22.158247 14901 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:37:22.158308 14901 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:37:22.158342 14901 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 18:37:22 dist-test-slave-j1xs krb5kdc[14784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707042, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.1@KRBTEST.COM for kudu/127.13.100.62@KRBTEST.COM
I20260929 18:37:22.169763 14901 rpc_server.cc:307] RPC server started. Bound to: 127.13.100.1:45983
I20260929 18:37:22.169848 15031 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.100.1:45983 every 8 connection(s)
I20260929 18:37:22.170825 14901 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707036041078-13712-0/minicluster-data/ts-0/data/info.pb
I20260929 18:37:22.172387 14878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:22.158997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.1:44553 (local address 127.13.100.62:34301)
0929 18:37:22.159168 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:22.159175 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:22.159945 (+ 770us) server_negotiation.cc:407] Connection header received
0929 18:37:22.160940 (+ 995us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:22.160943 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:22.161001 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:22.161071 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:22.163055 (+ 1984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:22.163633 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:22.164854 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:22.165158 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:22.165274 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:22.168345 (+ 3071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:22.168349 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:22.168352 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:37:22.168388 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:22.170071 (+ 1683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:22.170721 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:22.170725 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:22.170730 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:22.170824 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:22.171198 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:22.171204 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:22.171208 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:22.171452 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:22.171558 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:22.172062 (+ 504us) server_negotiation.cc:299] Negotiation successful
0929 18:37:22.172217 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 18:37:22.173228 14925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:22.159354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:34301 (local address 127.13.100.1:44553)
0929 18:37:22.159776 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:22.159825 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:22.160653 (+ 828us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:22.161568 (+ 915us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:22.161576 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:22.162211 (+ 635us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:37:22.162910 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:22.162935 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:22.163764 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:22.163767 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:22.164733 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:22.164743 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:22.165320 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:22.165324 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:22.165495 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:22.166200 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:37:22.166229 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:37:22.168163 (+ 1934us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:37:22.170260 (+ 2097us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:22.170265 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:22.170280 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:22.170576 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:22.170933 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:22.170936 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:22.170938 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:22.171094 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:22.171561 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:37:22.171567 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:37:22.171827 (+ 260us) client_negotiation.cc:769] Sending connection context
0929 18:37:22.172018 (+ 191us) client_negotiation.cc:240] Negotiation successful
0929 18:37:22.172335 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":117,"threads_started":1}
I20260929 18:37:22.174485 15032 heartbeater.cc:347] Connected to a master server at 127.13.100.62:34301
I20260929 18:37:22.174649 15032 heartbeater.cc:464] Registering TS with master...
I20260929 18:37:22.174741 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 14901
I20260929 18:37:22.174870 13712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707036041078-13712-0/minicluster-data/ts-0/wal/instance
I20260929 18:37:22.175164 13712 external_mini_cluster.cc:1428] Setting key 61e4a0ae4ad56fd1936030fcd475c896
I20260929 18:37:22.175189 15032 heartbeater.cc:511] Master 127.13.100.62:34301 requested a full tablet report, sending...
I20260929 18:37:22.176707 14829 ts_manager.cc:194] Registered new tserver with Master: fdfd0eb5ab40468594d7cb793150beb8 (127.13.100.1:45983)
I20260929 18:37:22.178426 14829 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.100.1@KRBTEST.COM'} at 127.13.100.1:44553
I20260929 18:37:22.184805 15038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:22.177058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:34301 (local address 127.13.100.1:35853)
0929 18:37:22.177322 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:22.177336 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:22.177455 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:22.177686 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:22.177689 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:22.177924 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:37:22.178039 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:22.178045 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:22.178949 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:22.178952 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:22.179640 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:22.179648 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:22.180138 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:22.180142 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:22.180240 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:22.180770 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:37:22.180782 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:37:22.181104 (+ 322us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:37:22.183370 (+ 2266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:22.183373 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:22.183374 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:22.183570 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:22.183870 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:22.183873 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:22.183874 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:22.183947 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:22.184379 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:37:22.184381 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:37:22.184457 (+ 76us) client_negotiation.cc:769] Sending connection context
0929 18:37:22.184536 (+ 79us) client_negotiation.cc:240] Negotiation successful
0929 18:37:22.184647 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":116,"threads_started":1}
I20260929 18:37:22.184943 14878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:22.176911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.1:35853 (local address 127.13.100.62:34301)
0929 18:37:22.177035 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:22.177040 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:22.177416 (+ 376us) server_negotiation.cc:407] Connection header received
0929 18:37:22.177529 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:22.177533 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:22.177573 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:22.177674 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:22.178145 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:22.178809 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:22.179766 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:22.180002 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:22.180113 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:22.181237 (+ 1124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:22.181241 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:22.181244 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:37:22.181266 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:22.183230 (+ 1964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:22.183672 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:22.183676 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:22.183678 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:22.183768 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:22.184034 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:22.184038 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:22.184040 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:22.184282 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:22.184394 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:22.184631 (+ 237us) server_negotiation.cc:299] Negotiation successful
0929 18:37:22.184753 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
WARNING: no policy specified for kudu/127.13.100.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.100.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.100.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.2.keytab.
Entry for principal kudu/127.13.100.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.2.keytab.
I20260929 18:37:22.204941 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707036041078-13712-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707036041078-13712-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707036041078-13712-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707036041078-13712-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.100.2:0
--local_ip_for_outbound_sockets=127.13.100.2
--webserver_interface=127.13.100.2
--webserver_port=0
--tserver_master_addrs=127.13.100.62:34301
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.2.keytab
--principal=kudu/127.13.100.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.100.20:38841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707036041078-13712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707036041078-13712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707036041078-13712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:37:22.314124 15041 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:37:22.314431 15041 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:37:22.314510 15041 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:37:22.318660 15041 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:37:22.318745 15041 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:37:22.318813 15041 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 18:37:22.318897 15041 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.100.2
I20260929 18:37:22.324546 15041 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.100.20:38841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707036041078-13712-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707036041078-13712-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.2.keytab
--principal=kudu/127.13.100.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.100.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707036041078-13712-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.100.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.13.100.62:34301
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15041
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.100.2
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707036041078-13712-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:37:22.325615 15041 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:37:22.326635 15041 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:37:22.336450 15046 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:37:22.336483 15047 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:37:22.336483 15049 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:37:22.336853 15041 server_base.cc:1034] running on GCE node
I20260929 18:37:22.337240 15041 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:37:22.337872 15041 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:37:22.339139 15041 hybrid_clock.cc:648] HybridClock initialized: now 1790707042339124 us; error 32 us; skew 500 ppm
Sep 29 18:37:22 dist-test-slave-j1xs krb5kdc[14784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707042, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:37:22.341953 15041 init.cc:377] Logged in from keytab as kudu/127.13.100.2@KRBTEST.COM (short username kudu)
I20260929 18:37:22.342824 15041 webserver.cc:460] Webserver started at http://127.13.100.2:43037/ using document root <none> and password file <none>
I20260929 18:37:22.343418 15041 fs_manager.cc:359] Metadata directory not provided
I20260929 18:37:22.343510 15041 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:37:22.343740 15041 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:37:22.345721 15041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707036041078-13712-0/minicluster-data/ts-1/data/instance:
uuid: "87b61322896e4e0c988d10faf801c9b5"
format_stamp: "Formatted at 2026-09-29 18:37:22 on dist-test-slave-j1xs"
server_key: "13aef2a367d601400f446b097890cb0b"
server_key_iv: "0f3502ab4deaeeba4a11617f60772ce0"
server_key_version: "encryptionkey@0"
I20260929 18:37:22.346328 15041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707036041078-13712-0/minicluster-data/ts-1/wal/instance:
uuid: "87b61322896e4e0c988d10faf801c9b5"
format_stamp: "Formatted at 2026-09-29 18:37:22 on dist-test-slave-j1xs"
server_key: "13aef2a367d601400f446b097890cb0b"
server_key_iv: "0f3502ab4deaeeba4a11617f60772ce0"
server_key_version: "encryptionkey@0"
I20260929 18:37:22.350127 15041 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 18:37:22.352417 15056 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:37:22.353646 15041 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:37:22.353772 15041 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707036041078-13712-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707036041078-13712-0/minicluster-data/ts-1/wal
uuid: "87b61322896e4e0c988d10faf801c9b5"
format_stamp: "Formatted at 2026-09-29 18:37:22 on dist-test-slave-j1xs"
server_key: "13aef2a367d601400f446b097890cb0b"
server_key_iv: "0f3502ab4deaeeba4a11617f60772ce0"
server_key_version: "encryptionkey@0"
I20260929 18:37:22.353876 15041 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707036041078-13712-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707036041078-13712-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707036041078-13712-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:37:22.370461 15041 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:37:22.373536 15041 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:37:22.373725 15041 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:37:22.375213 15041 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:37:22.375283 15041 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:37:22.375346 15041 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:37:22.375406 15041 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 18:37:22 dist-test-slave-j1xs krb5kdc[14784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707042, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.2@KRBTEST.COM for kudu/127.13.100.62@KRBTEST.COM
I20260929 18:37:22.387149 15041 rpc_server.cc:307] RPC server started. Bound to: 127.13.100.2:32859
I20260929 18:37:22.387266 15171 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.100.2:32859 every 8 connection(s)
I20260929 18:37:22.388129 15041 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707036041078-13712-0/minicluster-data/ts-1/data/info.pb
I20260929 18:37:22.390172 14878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:22.375997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.2:57213 (local address 127.13.100.62:34301)
0929 18:37:22.376144 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:22.376155 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:22.377099 (+ 944us) server_negotiation.cc:407] Connection header received
0929 18:37:22.378163 (+ 1064us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:22.378168 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:22.378227 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:22.378286 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:22.379805 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:22.381442 (+ 1637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:22.383214 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:22.383463 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:22.383585 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:22.386550 (+ 2965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:22.386553 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:22.386556 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:37:22.386580 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:22.388158 (+ 1578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:22.388707 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:22.388712 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:22.388715 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:22.388810 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:22.389194 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:22.389197 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:22.389198 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:22.389375 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:22.389453 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:22.389878 (+ 425us) server_negotiation.cc:299] Negotiation successful
0929 18:37:22.390054 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 18:37:22.390919 15065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:22.376504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:34301 (local address 127.13.100.2:57213)
0929 18:37:22.376948 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:22.376992 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:22.377921 (+ 929us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:22.378663 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:22.378674 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:22.379002 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:37:22.379577 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:22.379599 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:22.381660 (+ 2061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:22.381664 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:22.383091 (+ 1427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:22.383101 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:22.383763 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:22.383766 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:22.383902 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:22.384697 (+ 795us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:37:22.384738 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:37:22.386361 (+ 1623us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:37:22.388288 (+ 1927us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:22.388294 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:22.388308 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:22.388587 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:22.388915 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:22.388918 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:22.388920 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:22.389072 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:22.389464 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:37:22.389468 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:37:22.389675 (+ 207us) client_negotiation.cc:769] Sending connection context
0929 18:37:22.389815 (+ 140us) client_negotiation.cc:240] Negotiation successful
0929 18:37:22.390184 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":122,"threads_started":1}
I20260929 18:37:22.391446 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 15041
I20260929 18:37:22.391573 13712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707036041078-13712-0/minicluster-data/ts-1/wal/instance
I20260929 18:37:22.391896 13712 external_mini_cluster.cc:1428] Setting key 3984d8894dfc2b6a256e412352bae121
I20260929 18:37:22.392275 15172 heartbeater.cc:347] Connected to a master server at 127.13.100.62:34301
I20260929 18:37:22.392429 15172 heartbeater.cc:464] Registering TS with master...
I20260929 18:37:22.392944 15172 heartbeater.cc:511] Master 127.13.100.62:34301 requested a full tablet report, sending...
I20260929 18:37:22.393960 14828 ts_manager.cc:194] Registered new tserver with Master: 87b61322896e4e0c988d10faf801c9b5 (127.13.100.2:32859)
I20260929 18:37:22.394735 14828 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.100.2@KRBTEST.COM'} at 127.13.100.2:57213
WARNING: no policy specified for kudu/127.13.100.3@KRBTEST.COM; defaulting to no policy
I20260929 18:37:22.402802 15178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:22.394472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:34301 (local address 127.13.100.2:41711)
0929 18:37:22.395205 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:22.395222 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:22.395326 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:22.395587 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:22.395591 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:22.395832 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:37:22.395986 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:22.395997 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:22.396939 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:22.396945 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:22.397997 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:22.398009 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:22.398440 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:22.398446 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:22.398577 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:22.399061 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:37:22.399080 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:37:22.399554 (+ 474us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:37:22.401244 (+ 1690us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:22.401248 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:22.401250 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:22.401481 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:22.401779 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:22.401783 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:22.401785 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:22.401911 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:22.402324 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:37:22.402326 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:37:22.402412 (+ 86us) client_negotiation.cc:769] Sending connection context
0929 18:37:22.402507 (+ 95us) client_negotiation.cc:240] Negotiation successful
0929 18:37:22.402672 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":637,"thread_start_us":84,"threads_started":1}
I20260929 18:37:22.402998 14878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:22.394560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.2:41711 (local address 127.13.100.62:34301)
0929 18:37:22.394682 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:22.394689 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:22.395286 (+ 597us) server_negotiation.cc:407] Connection header received
0929 18:37:22.395419 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:22.395422 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:22.395467 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:22.395551 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:22.396125 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:22.396812 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:22.398116 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:22.398317 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:22.398395 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:22.399679 (+ 1284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:22.399682 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:22.399684 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:37:22.399704 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:22.401134 (+ 1430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:22.401578 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:22.401581 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:22.401582 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:22.401687 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:22.401993 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:22.401995 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:22.401996 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:22.402215 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:22.402293 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:22.402676 (+ 383us) server_negotiation.cc:299] Negotiation successful
0929 18:37:22.402803 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.100.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.100.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.3.keytab.
Entry for principal kudu/127.13.100.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.3.keytab.
I20260929 18:37:22.420681 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707036041078-13712-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707036041078-13712-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707036041078-13712-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707036041078-13712-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.100.3:0
--local_ip_for_outbound_sockets=127.13.100.3
--webserver_interface=127.13.100.3
--webserver_port=0
--tserver_master_addrs=127.13.100.62:34301
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.3.keytab
--principal=kudu/127.13.100.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.100.20:38841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707036041078-13712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707036041078-13712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707036041078-13712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:37:22.534723 15181 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:37:22.534963 15181 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:37:22.535006 15181 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:37:22.539305 15181 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:37:22.539361 15181 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:37:22.539400 15181 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 18:37:22.539443 15181 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.100.3
I20260929 18:37:22.544588 15181 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.100.20:38841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707036041078-13712-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707036041078-13712-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.3.keytab
--principal=kudu/127.13.100.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.100.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707036041078-13712-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.100.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.13.100.62:34301
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15181
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.100.3
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707036041078-13712-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:37:22.545516 15181 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:37:22.546550 15181 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:37:22.556167 15189 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:37:22.556340 15186 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:37:22.556263 15187 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:37:22.556535 15181 server_base.cc:1034] running on GCE node
I20260929 18:37:22.556896 15181 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:37:22.557471 15181 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:37:22.558640 15181 hybrid_clock.cc:648] HybridClock initialized: now 1790707042558638 us; error 35 us; skew 500 ppm
Sep 29 18:37:22 dist-test-slave-j1xs krb5kdc[14784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707042, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:37:22.561211 15181 init.cc:377] Logged in from keytab as kudu/127.13.100.3@KRBTEST.COM (short username kudu)
I20260929 18:37:22.562077 15181 webserver.cc:460] Webserver started at http://127.13.100.3:45655/ using document root <none> and password file <none>
I20260929 18:37:22.562604 15181 fs_manager.cc:359] Metadata directory not provided
I20260929 18:37:22.562664 15181 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:37:22.562817 15181 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:37:22.564832 15181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707036041078-13712-0/minicluster-data/ts-2/data/instance:
uuid: "945f5bbc44574d57a45b266d384d2c59"
format_stamp: "Formatted at 2026-09-29 18:37:22 on dist-test-slave-j1xs"
server_key: "d9d637d33afe8ba43497d868e5ef03de"
server_key_iv: "3b155d32a26a7c8a6eaddbbe5488d416"
server_key_version: "encryptionkey@0"
I20260929 18:37:22.565320 15181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707036041078-13712-0/minicluster-data/ts-2/wal/instance:
uuid: "945f5bbc44574d57a45b266d384d2c59"
format_stamp: "Formatted at 2026-09-29 18:37:22 on dist-test-slave-j1xs"
server_key: "d9d637d33afe8ba43497d868e5ef03de"
server_key_iv: "3b155d32a26a7c8a6eaddbbe5488d416"
server_key_version: "encryptionkey@0"
I20260929 18:37:22.568866 15181 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260929 18:37:22.571352 15196 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:37:22.572620 15181 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 18:37:22.572715 15181 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707036041078-13712-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707036041078-13712-0/minicluster-data/ts-2/wal
uuid: "945f5bbc44574d57a45b266d384d2c59"
format_stamp: "Formatted at 2026-09-29 18:37:22 on dist-test-slave-j1xs"
server_key: "d9d637d33afe8ba43497d868e5ef03de"
server_key_iv: "3b155d32a26a7c8a6eaddbbe5488d416"
server_key_version: "encryptionkey@0"
I20260929 18:37:22.572798 15181 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707036041078-13712-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707036041078-13712-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707036041078-13712-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:37:22.588951 15181 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:37:22.591979 15181 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:37:22.592190 15181 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:37:22.593729 15181 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:37:22.593802 15181 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:37:22.593875 15181 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:37:22.593955 15181 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
Sep 29 18:37:22 dist-test-slave-j1xs krb5kdc[14784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707042, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.3@KRBTEST.COM for kudu/127.13.100.62@KRBTEST.COM
I20260929 18:37:22.604980 15181 rpc_server.cc:307] RPC server started. Bound to: 127.13.100.3:32827
I20260929 18:37:22.605034 15311 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.100.3:32827 every 8 connection(s)
I20260929 18:37:22.606010 15181 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707036041078-13712-0/minicluster-data/ts-2/data/info.pb
I20260929 18:37:22.606993 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 15181
I20260929 18:37:22.607102 13712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707036041078-13712-0/minicluster-data/ts-2/wal/instance
I20260929 18:37:22.607421 13712 external_mini_cluster.cc:1428] Setting key f3fc1df910d4a18e1ebdf242cfc529f4
I20260929 18:37:22.609458 14878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:22.594478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.3:51349 (local address 127.13.100.62:34301)
0929 18:37:22.594610 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:22.594614 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:22.596282 (+ 1668us) server_negotiation.cc:407] Connection header received
0929 18:37:22.597458 (+ 1176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:22.597462 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:22.597523 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:22.597615 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:22.599315 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:22.600258 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:22.602193 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:22.602421 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:22.602515 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:22.605598 (+ 3083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:22.605601 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:22.605603 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:37:22.605629 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:22.607304 (+ 1675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:22.607845 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:22.607849 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:22.607851 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:22.607932 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:22.608310 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:22.608315 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:22.608317 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:22.608497 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:22.608605 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:22.609175 (+ 570us) server_negotiation.cc:299] Negotiation successful
0929 18:37:22.609288 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 18:37:22.610319 15205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:22.595531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:34301 (local address 127.13.100.3:51349)
0929 18:37:22.596109 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:22.596156 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:22.597230 (+ 1074us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:22.597811 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:22.597823 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:22.598372 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:37:22.599121 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:22.599144 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:22.600427 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:22.600433 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:22.602025 (+ 1592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:22.602039 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:22.602596 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:22.602601 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:22.602821 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:22.603598 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:37:22.603640 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:37:22.605423 (+ 1783us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:37:22.607418 (+ 1995us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:22.607425 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:22.607440 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:22.607719 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:22.608031 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:22.608035 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:22.608037 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:22.608203 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:22.608608 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:37:22.608617 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:37:22.608883 (+ 266us) client_negotiation.cc:769] Sending connection context
0929 18:37:22.609086 (+ 203us) client_negotiation.cc:240] Negotiation successful
0929 18:37:22.609405 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":125,"threads_started":1}
I20260929 18:37:22.611493 15312 heartbeater.cc:347] Connected to a master server at 127.13.100.62:34301
I20260929 18:37:22.611630 15312 heartbeater.cc:464] Registering TS with master...
I20260929 18:37:22.612165 15312 heartbeater.cc:511] Master 127.13.100.62:34301 requested a full tablet report, sending...
I20260929 18:37:22.613174 14828 ts_manager.cc:194] Registered new tserver with Master: 945f5bbc44574d57a45b266d384d2c59 (127.13.100.3:32827)
I20260929 18:37:22.613945 14828 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.100.3@KRBTEST.COM'} at 127.13.100.3:51349
I20260929 18:37:22.620680 14878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:22.613844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.3:53837 (local address 127.13.100.62:34301)
0929 18:37:22.613972 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:22.613975 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:22.614222 (+ 247us) server_negotiation.cc:407] Connection header received
0929 18:37:22.614375 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:22.614378 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:22.614421 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:22.614510 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:22.615037 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:22.615523 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:22.616438 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:22.616646 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:22.616736 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:22.617758 (+ 1022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:22.617762 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:22.617764 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:37:22.617783 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:22.619047 (+ 1264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:22.619510 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:22.619512 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:22.619514 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:22.619584 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:22.619828 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:22.619830 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:22.619832 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:22.620018 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:22.620118 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:22.620437 (+ 319us) server_negotiation.cc:299] Negotiation successful
0929 18:37:22.620539 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 18:37:22.620697 15318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:22.613759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:34301 (local address 127.13.100.3:53837)
0929 18:37:22.614127 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:22.614140 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:22.614274 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:22.614551 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:22.614554 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:22.614780 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:37:22.614930 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:22.614938 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:22.615653 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:22.615657 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:22.616318 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:22.616326 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:22.616761 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:22.616763 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:22.616830 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:22.617333 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:37:22.617349 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:37:22.617646 (+ 297us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:37:22.619161 (+ 1515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:22.619166 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:22.619168 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:22.619429 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:22.619662 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:22.619664 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:22.619665 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:22.619735 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:22.620133 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:37:22.620137 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:37:22.620249 (+ 112us) client_negotiation.cc:769] Sending connection context
0929 18:37:22.620353 (+ 104us) client_negotiation.cc:240] Negotiation successful
0929 18:37:22.620524 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"mutex_wait_us":3,"thread_start_us":90,"threads_started":1}
I20260929 18:37:22.621569 13712 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 18:37:22 dist-test-slave-j1xs krb5kdc[14784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707042, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 18:37:22 dist-test-slave-j1xs krb5kdc[14784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707042, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.13.100.62@KRBTEST.COM
I20260929 18:37:22.645620 14878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:22.637852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48600 (local address 127.13.100.62:34301)
0929 18:37:22.638010 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:22.638013 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:22.638024 (+ 11us) server_negotiation.cc:407] Connection header received
0929 18:37:22.638058 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:22.638060 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:22.638094 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:22.638157 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:22.638820 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:22.639402 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:22.640231 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:22.640516 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:22.640596 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:22.642999 (+ 2403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:22.643002 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:22.643004 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:37:22.643025 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:22.644274 (+ 1249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:22.644683 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:22.644686 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:22.644687 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:22.644754 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:22.644975 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:22.644976 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:22.644978 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:22.645128 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:22.645238 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:22.645421 (+ 183us) server_negotiation.cc:299] Negotiation successful
0929 18:37:22.645507 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260929 18:37:22.648588 14828 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48600:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:37:22.651641 14828 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:37:22.665997 15330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:22.659806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.2:32859 (local address 127.0.0.1:40824)
0929 18:37:22.660402 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:22.660449 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:22.660580 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:22.661869 (+ 1289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:22.661872 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:22.661911 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:37:22.662100 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:22.662105 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:22.663402 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:22.663405 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:22.664647 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:22.664656 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:22.665423 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:22.665426 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:22.665544 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:22.665589 (+ 45us) client_negotiation.cc:769] Sending connection context
0929 18:37:22.665692 (+ 103us) client_negotiation.cc:240] Negotiation successful
0929 18:37:22.665809 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433,"mutex_wait_us":113,"spinlock_wait_cycles":12288,"thread_start_us":71,"threads_started":1}
I20260929 18:37:22.666046 15329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:22.659871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.3:32827 (local address 127.0.0.1:45968)
0929 18:37:22.660404 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:22.660449 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:22.660581 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:22.661716 (+ 1135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:22.661722 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:22.661744 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:37:22.661879 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:22.661913 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:22.663307 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:22.663313 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:22.664691 (+ 1378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:22.664700 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:22.665689 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:22.665692 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:22.665781 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:22.665799 (+ 18us) client_negotiation.cc:769] Sending connection context
0929 18:37:22.665856 (+ 57us) client_negotiation.cc:240] Negotiation successful
0929 18:37:22.665935 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"mutex_wait_us":94,"spinlock_wait_cycles":7040,"thread_start_us":93,"threads_started":1}
I20260929 18:37:22.666082 15334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:22.661003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53896 (local address 127.13.100.1:45983)
0929 18:37:22.661275 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:22.661290 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:22.661311 (+ 21us) server_negotiation.cc:407] Connection header received
0929 18:37:22.661387 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:22.661392 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:22.661586 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:22.661733 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:37:22.662234 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:22.662984 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:22.664390 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:22.665096 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:22.665190 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:22.665342 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:22.665804 (+ 462us) server_negotiation.cc:299] Negotiation successful
0929 18:37:22.665944 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":78,"threads_started":1}
I20260929 18:37:22.666133 15333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:22.660838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45968 (local address 127.13.100.3:32827)
0929 18:37:22.661341 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:22.661349 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:22.661367 (+ 18us) server_negotiation.cc:407] Connection header received
0929 18:37:22.661424 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:22.661428 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:22.661563 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:22.661727 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:37:22.662003 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:22.663177 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:22.664819 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:22.665548 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:22.665692 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:22.665835 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:22.665879 (+ 44us) server_negotiation.cc:299] Negotiation successful
0929 18:37:22.665994 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":417,"thread_start_us":116,"threads_started":1}
I20260929 18:37:22.666409 15332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:22.660474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.1:45983 (local address 127.0.0.1:53896)
0929 18:37:22.661018 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:22.661028 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:22.661128 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:22.661869 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:22.661872 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:22.661924 (+ 52us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:37:22.662096 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:22.662103 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:22.663118 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:22.663122 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:22.664263 (+ 1141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:22.664272 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:22.665402 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:22.665405 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:22.665539 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:22.665587 (+ 48us) client_negotiation.cc:769] Sending connection context
0929 18:37:22.665687 (+ 100us) client_negotiation.cc:240] Negotiation successful
0929 18:37:22.666164 (+ 477us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"mutex_wait_us":405,"spinlock_wait_cycles":67968,"thread_start_us":82,"threads_started":1}
I20260929 18:37:22.666810 15331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:22.659908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40824 (local address 127.13.100.2:32859)
0929 18:37:22.661346 (+ 1438us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:22.661357 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:22.661374 (+ 17us) server_negotiation.cc:407] Connection header received
0929 18:37:22.661432 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:22.661436 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:22.661589 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:22.661733 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:37:22.662227 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:22.663283 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:22.664773 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:22.665302 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:22.665425 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:22.665582 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:22.666548 (+ 966us) server_negotiation.cc:299] Negotiation successful
0929 18:37:22.666664 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1349,"thread_start_us":107,"threads_started":1}
I20260929 18:37:22.669579 14966 tablet_service.cc:1467] Processing CreateTablet for tablet 6def05ee84cb41aeb511d2517914e9d3 (DEFAULT_TABLE table=test-table [id=ada2a9ea621d4604a9c7897fa7c89990]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:37:22.669569 15106 tablet_service.cc:1467] Processing CreateTablet for tablet 6def05ee84cb41aeb511d2517914e9d3 (DEFAULT_TABLE table=test-table [id=ada2a9ea621d4604a9c7897fa7c89990]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:37:22.669675 15246 tablet_service.cc:1467] Processing CreateTablet for tablet 6def05ee84cb41aeb511d2517914e9d3 (DEFAULT_TABLE table=test-table [id=ada2a9ea621d4604a9c7897fa7c89990]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:37:22.670722 15106 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6def05ee84cb41aeb511d2517914e9d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:37:22.670722 14966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6def05ee84cb41aeb511d2517914e9d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:37:22.670866 15246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6def05ee84cb41aeb511d2517914e9d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:37:22.677484 15335 tablet_bootstrap.cc:492] T 6def05ee84cb41aeb511d2517914e9d3 P 945f5bbc44574d57a45b266d384d2c59: Bootstrap starting.
I20260929 18:37:22.678196 15337 tablet_bootstrap.cc:492] T 6def05ee84cb41aeb511d2517914e9d3 P 87b61322896e4e0c988d10faf801c9b5: Bootstrap starting.
I20260929 18:37:22.680075 15337 tablet_bootstrap.cc:654] T 6def05ee84cb41aeb511d2517914e9d3 P 87b61322896e4e0c988d10faf801c9b5: Neither blocks nor log segments found. Creating new log.
I20260929 18:37:22.680153 15335 tablet_bootstrap.cc:654] T 6def05ee84cb41aeb511d2517914e9d3 P 945f5bbc44574d57a45b266d384d2c59: Neither blocks nor log segments found. Creating new log.
I20260929 18:37:22.680807 15337 log.cc:824] T 6def05ee84cb41aeb511d2517914e9d3 P 87b61322896e4e0c988d10faf801c9b5: Log is configured to *not* fsync() on all Append() calls
I20260929 18:37:22.680811 15335 log.cc:824] T 6def05ee84cb41aeb511d2517914e9d3 P 945f5bbc44574d57a45b266d384d2c59: Log is configured to *not* fsync() on all Append() calls
I20260929 18:37:22.681427 15336 tablet_bootstrap.cc:492] T 6def05ee84cb41aeb511d2517914e9d3 P fdfd0eb5ab40468594d7cb793150beb8: Bootstrap starting.
I20260929 18:37:22.682976 15335 tablet_bootstrap.cc:492] T 6def05ee84cb41aeb511d2517914e9d3 P 945f5bbc44574d57a45b266d384d2c59: No bootstrap required, opened a new log
I20260929 18:37:22.682974 15337 tablet_bootstrap.cc:492] T 6def05ee84cb41aeb511d2517914e9d3 P 87b61322896e4e0c988d10faf801c9b5: No bootstrap required, opened a new log
I20260929 18:37:22.683192 15335 ts_tablet_manager.cc:1397] T 6def05ee84cb41aeb511d2517914e9d3 P 945f5bbc44574d57a45b266d384d2c59: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 18:37:22.683192 15337 ts_tablet_manager.cc:1397] T 6def05ee84cb41aeb511d2517914e9d3 P 87b61322896e4e0c988d10faf801c9b5: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 18:37:22.683770 15336 tablet_bootstrap.cc:654] T 6def05ee84cb41aeb511d2517914e9d3 P fdfd0eb5ab40468594d7cb793150beb8: Neither blocks nor log segments found. Creating new log.
I20260929 18:37:22.684365 15336 log.cc:824] T 6def05ee84cb41aeb511d2517914e9d3 P fdfd0eb5ab40468594d7cb793150beb8: Log is configured to *not* fsync() on all Append() calls
I20260929 18:37:22.686067 15336 tablet_bootstrap.cc:492] T 6def05ee84cb41aeb511d2517914e9d3 P fdfd0eb5ab40468594d7cb793150beb8: No bootstrap required, opened a new log
I20260929 18:37:22.686223 15336 ts_tablet_manager.cc:1397] T 6def05ee84cb41aeb511d2517914e9d3 P fdfd0eb5ab40468594d7cb793150beb8: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 18:37:22.687574 15337 raft_consensus.cc:348] T 6def05ee84cb41aeb511d2517914e9d3 P 87b61322896e4e0c988d10faf801c9b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87b61322896e4e0c988d10faf801c9b5" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 32859 } } peers { permanent_uuid: "945f5bbc44574d57a45b266d384d2c59" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 32827 } } peers { permanent_uuid: "fdfd0eb5ab40468594d7cb793150beb8" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 45983 } }
I20260929 18:37:22.687671 15335 raft_consensus.cc:348] T 6def05ee84cb41aeb511d2517914e9d3 P 945f5bbc44574d57a45b266d384d2c59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87b61322896e4e0c988d10faf801c9b5" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 32859 } } peers { permanent_uuid: "945f5bbc44574d57a45b266d384d2c59" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 32827 } } peers { permanent_uuid: "fdfd0eb5ab40468594d7cb793150beb8" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 45983 } }
I20260929 18:37:22.687896 15335 raft_consensus.cc:374] T 6def05ee84cb41aeb511d2517914e9d3 P 945f5bbc44574d57a45b266d384d2c59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:37:22.687896 15337 raft_consensus.cc:374] T 6def05ee84cb41aeb511d2517914e9d3 P 87b61322896e4e0c988d10faf801c9b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:37:22.687959 15337 raft_consensus.cc:729] T 6def05ee84cb41aeb511d2517914e9d3 P 87b61322896e4e0c988d10faf801c9b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87b61322896e4e0c988d10faf801c9b5, State: Initialized, Role: FOLLOWER
I20260929 18:37:22.687959 15335 raft_consensus.cc:729] T 6def05ee84cb41aeb511d2517914e9d3 P 945f5bbc44574d57a45b266d384d2c59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 945f5bbc44574d57a45b266d384d2c59, State: Initialized, Role: FOLLOWER
I20260929 18:37:22.688444 15337 consensus_queue.cc:260] T 6def05ee84cb41aeb511d2517914e9d3 P 87b61322896e4e0c988d10faf801c9b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87b61322896e4e0c988d10faf801c9b5" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 32859 } } peers { permanent_uuid: "945f5bbc44574d57a45b266d384d2c59" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 32827 } } peers { permanent_uuid: "fdfd0eb5ab40468594d7cb793150beb8" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 45983 } }
I20260929 18:37:22.688444 15335 consensus_queue.cc:260] T 6def05ee84cb41aeb511d2517914e9d3 P 945f5bbc44574d57a45b266d384d2c59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87b61322896e4e0c988d10faf801c9b5" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 32859 } } peers { permanent_uuid: "945f5bbc44574d57a45b266d384d2c59" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 32827 } } peers { permanent_uuid: "fdfd0eb5ab40468594d7cb793150beb8" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 45983 } }
I20260929 18:37:22.689558 15335 ts_tablet_manager.cc:1428] T 6def05ee84cb41aeb511d2517914e9d3 P 945f5bbc44574d57a45b266d384d2c59: Time spent starting tablet: real 0.006s user 0.000s sys 0.005s
I20260929 18:37:22.689548 15337 ts_tablet_manager.cc:1428] T 6def05ee84cb41aeb511d2517914e9d3 P 87b61322896e4e0c988d10faf801c9b5: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260929 18:37:22.689461 15336 raft_consensus.cc:348] T 6def05ee84cb41aeb511d2517914e9d3 P fdfd0eb5ab40468594d7cb793150beb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87b61322896e4e0c988d10faf801c9b5" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 32859 } } peers { permanent_uuid: "945f5bbc44574d57a45b266d384d2c59" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 32827 } } peers { permanent_uuid: "fdfd0eb5ab40468594d7cb793150beb8" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 45983 } }
I20260929 18:37:22.689729 15336 raft_consensus.cc:374] T 6def05ee84cb41aeb511d2517914e9d3 P fdfd0eb5ab40468594d7cb793150beb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:37:22.689953 15312 heartbeater.cc:503] Master 127.13.100.62:34301 was elected leader, sending a full tablet report...
I20260929 18:37:22.689953 15172 heartbeater.cc:503] Master 127.13.100.62:34301 was elected leader, sending a full tablet report...
I20260929 18:37:22.690882 15336 raft_consensus.cc:729] T 6def05ee84cb41aeb511d2517914e9d3 P fdfd0eb5ab40468594d7cb793150beb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdfd0eb5ab40468594d7cb793150beb8, State: Initialized, Role: FOLLOWER
I20260929 18:37:22.691360 15336 consensus_queue.cc:260] T 6def05ee84cb41aeb511d2517914e9d3 P fdfd0eb5ab40468594d7cb793150beb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87b61322896e4e0c988d10faf801c9b5" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 32859 } } peers { permanent_uuid: "945f5bbc44574d57a45b266d384d2c59" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 32827 } } peers { permanent_uuid: "fdfd0eb5ab40468594d7cb793150beb8" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 45983 } }
I20260929 18:37:22.691627 15341 raft_consensus.cc:482] T 6def05ee84cb41aeb511d2517914e9d3 P 945f5bbc44574d57a45b266d384d2c59 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:37:22.691756 15341 raft_consensus.cc:504] T 6def05ee84cb41aeb511d2517914e9d3 P 945f5bbc44574d57a45b266d384d2c59 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87b61322896e4e0c988d10faf801c9b5" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 32859 } } peers { permanent_uuid: "945f5bbc44574d57a45b266d384d2c59" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 32827 } } peers { permanent_uuid: "fdfd0eb5ab40468594d7cb793150beb8" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 45983 } }
I20260929 18:37:22.692391 15336 ts_tablet_manager.cc:1428] T 6def05ee84cb41aeb511d2517914e9d3 P fdfd0eb5ab40468594d7cb793150beb8: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260929 18:37:22.692744 15032 heartbeater.cc:503] Master 127.13.100.62:34301 was elected leader, sending a full tablet report...
I20260929 18:37:22.694343 15341 leader_election.cc:288] T 6def05ee84cb41aeb511d2517914e9d3 P 945f5bbc44574d57a45b266d384d2c59 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87b61322896e4e0c988d10faf801c9b5 (127.13.100.2:32859), fdfd0eb5ab40468594d7cb793150beb8 (127.13.100.1:45983)
I20260929 18:37:22.698417 15331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:22.693586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.3:34941 (local address 127.13.100.2:32859)
0929 18:37:22.693720 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:22.693724 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:22.693737 (+ 13us) server_negotiation.cc:407] Connection header received
0929 18:37:22.693777 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:22.693781 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:22.693824 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:22.694017 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:37:22.694942 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:22.695714 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:22.697030 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:22.697557 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:22.697706 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:22.697753 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:22.698250 (+ 497us) server_negotiation.cc:299] Negotiation successful
0929 18:37:22.698316 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 18:37:22.698413 15205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:22.693078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.2:32859 (local address 127.13.100.3:34941)
0929 18:37:22.693333 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:22.693349 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:22.693425 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:22.693959 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:22.693962 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:22.693982 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:37:22.694714 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:22.694720 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:22.695832 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:22.695835 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:22.696913 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:22.696921 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:22.697874 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:22.697883 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:22.698079 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:22.698116 (+ 37us) client_negotiation.cc:769] Sending connection context
0929 18:37:22.698201 (+ 85us) client_negotiation.cc:240] Negotiation successful
0929 18:37:22.698270 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"mutex_wait_us":597}
I20260929 18:37:22.699029 15113 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6def05ee84cb41aeb511d2517914e9d3" candidate_uuid: "945f5bbc44574d57a45b266d384d2c59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87b61322896e4e0c988d10faf801c9b5" is_pre_election: true
I20260929 18:37:22.699359 15113 raft_consensus.cc:2454] T 6def05ee84cb41aeb511d2517914e9d3 P 87b61322896e4e0c988d10faf801c9b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 945f5bbc44574d57a45b266d384d2c59 in term 0.
I20260929 18:37:22.699970 15199 leader_election.cc:302] T 6def05ee84cb41aeb511d2517914e9d3 P 945f5bbc44574d57a45b266d384d2c59 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87b61322896e4e0c988d10faf801c9b5, 945f5bbc44574d57a45b266d384d2c59; no voters:
I20260929 18:37:22.700232 15334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:22.694216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.3:50809 (local address 127.13.100.1:45983)
0929 18:37:22.694380 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:22.694385 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:22.695428 (+ 1043us) server_negotiation.cc:407] Connection header received
0929 18:37:22.695584 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:22.695587 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:22.695634 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:22.695697 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:37:22.696537 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:22.697079 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:22.698964 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:22.699445 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:22.699581 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:22.699632 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:22.699971 (+ 339us) server_negotiation.cc:299] Negotiation successful
0929 18:37:22.700035 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260929 18:37:22.700299 15341 raft_consensus.cc:2781] T 6def05ee84cb41aeb511d2517914e9d3 P 945f5bbc44574d57a45b266d384d2c59 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:37:22.700398 15341 raft_consensus.cc:482] T 6def05ee84cb41aeb511d2517914e9d3 P 945f5bbc44574d57a45b266d384d2c59 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:37:22.700443 15344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:22.694110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.1:45983 (local address 127.13.100.3:50809)
0929 18:37:22.695346 (+ 1236us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:22.695360 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:22.695485 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:22.696263 (+ 778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:22.696266 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:22.696284 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:37:22.696402 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:22.696411 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:22.697249 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:22.697251 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:22.698845 (+ 1594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:22.698854 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:22.699579 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:22.699583 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:22.699824 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:22.699849 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 18:37:22.699930 (+ 81us) client_negotiation.cc:240] Negotiation successful
0929 18:37:22.700010 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1174,"thread_start_us":115,"threads_started":1}
I20260929 18:37:22.700450 15341 raft_consensus.cc:3037] T 6def05ee84cb41aeb511d2517914e9d3 P 945f5bbc44574d57a45b266d384d2c59 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:37:22.701040 14986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6def05ee84cb41aeb511d2517914e9d3" candidate_uuid: "945f5bbc44574d57a45b266d384d2c59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdfd0eb5ab40468594d7cb793150beb8" is_pre_election: true
I20260929 18:37:22.701351 14986 raft_consensus.cc:2454] T 6def05ee84cb41aeb511d2517914e9d3 P fdfd0eb5ab40468594d7cb793150beb8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 945f5bbc44574d57a45b266d384d2c59 in term 0.
I20260929 18:37:22.701795 15341 raft_consensus.cc:504] T 6def05ee84cb41aeb511d2517914e9d3 P 945f5bbc44574d57a45b266d384d2c59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87b61322896e4e0c988d10faf801c9b5" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 32859 } } peers { permanent_uuid: "945f5bbc44574d57a45b266d384d2c59" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 32827 } } peers { permanent_uuid: "fdfd0eb5ab40468594d7cb793150beb8" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 45983 } }
I20260929 18:37:22.702165 15341 leader_election.cc:288] T 6def05ee84cb41aeb511d2517914e9d3 P 945f5bbc44574d57a45b266d384d2c59 [CANDIDATE]: Term 1 election: Requested vote from peers 87b61322896e4e0c988d10faf801c9b5 (127.13.100.2:32859), fdfd0eb5ab40468594d7cb793150beb8 (127.13.100.1:45983)
I20260929 18:37:22.702682 15113 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6def05ee84cb41aeb511d2517914e9d3" candidate_uuid: "945f5bbc44574d57a45b266d384d2c59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87b61322896e4e0c988d10faf801c9b5"
I20260929 18:37:22.702682 14986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6def05ee84cb41aeb511d2517914e9d3" candidate_uuid: "945f5bbc44574d57a45b266d384d2c59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdfd0eb5ab40468594d7cb793150beb8"
I20260929 18:37:22.703015 15113 raft_consensus.cc:3037] T 6def05ee84cb41aeb511d2517914e9d3 P 87b61322896e4e0c988d10faf801c9b5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:37:22.703014 14986 raft_consensus.cc:3037] T 6def05ee84cb41aeb511d2517914e9d3 P fdfd0eb5ab40468594d7cb793150beb8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:37:22.704517 14986 raft_consensus.cc:2454] T 6def05ee84cb41aeb511d2517914e9d3 P fdfd0eb5ab40468594d7cb793150beb8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 945f5bbc44574d57a45b266d384d2c59 in term 1.
I20260929 18:37:22.704819 15113 raft_consensus.cc:2454] T 6def05ee84cb41aeb511d2517914e9d3 P 87b61322896e4e0c988d10faf801c9b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 945f5bbc44574d57a45b266d384d2c59 in term 1.
I20260929 18:37:22.704859 15198 leader_election.cc:302] T 6def05ee84cb41aeb511d2517914e9d3 P 945f5bbc44574d57a45b266d384d2c59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 945f5bbc44574d57a45b266d384d2c59, fdfd0eb5ab40468594d7cb793150beb8; no voters:
I20260929 18:37:22.705013 15341 raft_consensus.cc:2781] T 6def05ee84cb41aeb511d2517914e9d3 P 945f5bbc44574d57a45b266d384d2c59 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:37:22.705245 15341 raft_consensus.cc:686] T 6def05ee84cb41aeb511d2517914e9d3 P 945f5bbc44574d57a45b266d384d2c59 [term 1 LEADER]: Becoming Leader. State: Replica: 945f5bbc44574d57a45b266d384d2c59, State: Running, Role: LEADER
I20260929 18:37:22.705598 15341 consensus_queue.cc:237] T 6def05ee84cb41aeb511d2517914e9d3 P 945f5bbc44574d57a45b266d384d2c59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87b61322896e4e0c988d10faf801c9b5" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 32859 } } peers { permanent_uuid: "945f5bbc44574d57a45b266d384d2c59" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 32827 } } peers { permanent_uuid: "fdfd0eb5ab40468594d7cb793150beb8" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 45983 } }
I20260929 18:37:22.708937 14828 catalog_manager.cc:5495] T 6def05ee84cb41aeb511d2517914e9d3 P 945f5bbc44574d57a45b266d384d2c59 reported cstate change: term changed from 0 to 1, leader changed from <none> to 945f5bbc44574d57a45b266d384d2c59 (127.13.100.3). New cstate: current_term: 1 leader_uuid: "945f5bbc44574d57a45b266d384d2c59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87b61322896e4e0c988d10faf801c9b5" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 32859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "945f5bbc44574d57a45b266d384d2c59" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 32827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fdfd0eb5ab40468594d7cb793150beb8" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 45983 } health_report { overall_health: UNKNOWN } } }
I20260929 18:37:22.730119 15333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:22.726037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45980 (local address 127.13.100.3:32827)
0929 18:37:22.726187 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:22.726190 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:22.726207 (+ 17us) server_negotiation.cc:407] Connection header received
0929 18:37:22.726353 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:22.726356 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:22.726654 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:22.726806 (+ 152us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:37:22.727151 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:22.727682 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:22.728603 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:22.728817 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:22.728900 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:22.729172 (+ 272us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:37:22.729584 (+ 412us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:37:22.729670 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:22.729809 (+ 139us) server_negotiation.cc:299] Negotiation successful
0929 18:37:22.729857 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 18:37:22.735836 15113 raft_consensus.cc:1261] T 6def05ee84cb41aeb511d2517914e9d3 P 87b61322896e4e0c988d10faf801c9b5 [term 1 FOLLOWER]: Refusing update from remote peer 945f5bbc44574d57a45b266d384d2c59: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:37:22.735836 14986 raft_consensus.cc:1261] T 6def05ee84cb41aeb511d2517914e9d3 P fdfd0eb5ab40468594d7cb793150beb8 [term 1 FOLLOWER]: Refusing update from remote peer 945f5bbc44574d57a45b266d384d2c59: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:37:22.736519 15341 consensus_queue.cc:1035] T 6def05ee84cb41aeb511d2517914e9d3 P 945f5bbc44574d57a45b266d384d2c59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87b61322896e4e0c988d10faf801c9b5" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 32859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:37:22.736753 15345 consensus_queue.cc:1035] T 6def05ee84cb41aeb511d2517914e9d3 P 945f5bbc44574d57a45b266d384d2c59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfd0eb5ab40468594d7cb793150beb8" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 45983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:37:22.745301 15349 mvcc.cc:205] Tried to move back new op lower bound from 7334736047038980096 to 7334736046925185024. Current Snapshot: MvccSnapshot[applied={T|T < 7334736047038980096}]
I20260929 18:37:22.757977 14829 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48600:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:37:22.758198 14829 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48600:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:37:22.760696 14829 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8da0f949c56c4288b898ad0637fee260: Sending DeleteTablet for 3 replicas of tablet 6def05ee84cb41aeb511d2517914e9d3
I20260929 18:37:22.761554 15246 tablet_service.cc:1514] Processing DeleteTablet for tablet 6def05ee84cb41aeb511d2517914e9d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:37:22 UTC) from {username='kudu', principal='kudu/127.13.100.62@KRBTEST.COM'} at 127.0.0.1:45968
I20260929 18:37:22.761557 15106 tablet_service.cc:1514] Processing DeleteTablet for tablet 6def05ee84cb41aeb511d2517914e9d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:37:22 UTC) from {username='kudu', principal='kudu/127.13.100.62@KRBTEST.COM'} at 127.0.0.1:40824
I20260929 18:37:22.762243 14966 tablet_service.cc:1514] Processing DeleteTablet for tablet 6def05ee84cb41aeb511d2517914e9d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:37:22 UTC) from {username='kudu', principal='kudu/127.13.100.62@KRBTEST.COM'} at 127.0.0.1:53896
I20260929 18:37:22.762315 15359 tablet_replica.cc:330] T 6def05ee84cb41aeb511d2517914e9d3 P 87b61322896e4e0c988d10faf801c9b5: stopping tablet replica
I20260929 18:37:22.762246 14829 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48600:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:37:22.762653 14829 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:37:22.762717 15360 tablet_replica.cc:330] T 6def05ee84cb41aeb511d2517914e9d3 P fdfd0eb5ab40468594d7cb793150beb8: stopping tablet replica
I20260929 18:37:22.762909 15359 raft_consensus.cc:2229] T 6def05ee84cb41aeb511d2517914e9d3 P 87b61322896e4e0c988d10faf801c9b5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:37:22.763005 15360 raft_consensus.cc:2229] T 6def05ee84cb41aeb511d2517914e9d3 P fdfd0eb5ab40468594d7cb793150beb8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:37:22.763310 15359 raft_consensus.cc:2258] T 6def05ee84cb41aeb511d2517914e9d3 P 87b61322896e4e0c988d10faf801c9b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:37:22.763309 15360 raft_consensus.cc:2258] T 6def05ee84cb41aeb511d2517914e9d3 P fdfd0eb5ab40468594d7cb793150beb8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:37:22.764925 15359 ts_tablet_manager.cc:1905] T 6def05ee84cb41aeb511d2517914e9d3 P 87b61322896e4e0c988d10faf801c9b5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:37:22.765087 15360 ts_tablet_manager.cc:1905] T 6def05ee84cb41aeb511d2517914e9d3 P fdfd0eb5ab40468594d7cb793150beb8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:37:22.767683 15359 ts_tablet_manager.cc:1918] T 6def05ee84cb41aeb511d2517914e9d3 P 87b61322896e4e0c988d10faf801c9b5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:37:22.768098 15358 tablet_replica.cc:330] T 6def05ee84cb41aeb511d2517914e9d3 P 945f5bbc44574d57a45b266d384d2c59: stopping tablet replica
I20260929 18:37:22.767729 15106 tablet_service.cc:1467] Processing CreateTablet for tablet cbccb7dca15c4bdfa1618d176011a9aa (DEFAULT_TABLE table=test-table [id=c1e847bffa5a4175972ab0a3e0ff0243]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:37:22.768378 15106 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cbccb7dca15c4bdfa1618d176011a9aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:37:22.767380 14966 tablet_service.cc:1467] Processing CreateTablet for tablet cbccb7dca15c4bdfa1618d176011a9aa (DEFAULT_TABLE table=test-table [id=c1e847bffa5a4175972ab0a3e0ff0243]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:37:22.768646 14966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cbccb7dca15c4bdfa1618d176011a9aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:37:22.770499 15337 tablet_bootstrap.cc:492] T cbccb7dca15c4bdfa1618d176011a9aa P 87b61322896e4e0c988d10faf801c9b5: Bootstrap starting.
I20260929 18:37:22.770694 15359 log.cc:1196] T 6def05ee84cb41aeb511d2517914e9d3 P 87b61322896e4e0c988d10faf801c9b5: Deleting WAL directory at /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707036041078-13712-0/minicluster-data/ts-1/wal/wals/6def05ee84cb41aeb511d2517914e9d3
I20260929 18:37:22.768275 15246 tablet_service.cc:1467] Processing CreateTablet for tablet cbccb7dca15c4bdfa1618d176011a9aa (DEFAULT_TABLE table=test-table [id=c1e847bffa5a4175972ab0a3e0ff0243]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:37:22.770960 15246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cbccb7dca15c4bdfa1618d176011a9aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:37:22.771618 15336 tablet_bootstrap.cc:492] T cbccb7dca15c4bdfa1618d176011a9aa P fdfd0eb5ab40468594d7cb793150beb8: Bootstrap starting.
I20260929 18:37:22.772569 15336 tablet_bootstrap.cc:654] T cbccb7dca15c4bdfa1618d176011a9aa P fdfd0eb5ab40468594d7cb793150beb8: Neither blocks nor log segments found. Creating new log.
I20260929 18:37:22.771620 15360 ts_tablet_manager.cc:1918] T 6def05ee84cb41aeb511d2517914e9d3 P fdfd0eb5ab40468594d7cb793150beb8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:37:22.773144 15360 log.cc:1196] T 6def05ee84cb41aeb511d2517914e9d3 P fdfd0eb5ab40468594d7cb793150beb8: Deleting WAL directory at /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707036041078-13712-0/minicluster-data/ts-0/wal/wals/6def05ee84cb41aeb511d2517914e9d3
I20260929 18:37:22.773327 15335 tablet_bootstrap.cc:492] T cbccb7dca15c4bdfa1618d176011a9aa P 945f5bbc44574d57a45b266d384d2c59: Bootstrap starting.
I20260929 18:37:22.773680 15336 tablet_bootstrap.cc:492] T cbccb7dca15c4bdfa1618d176011a9aa P fdfd0eb5ab40468594d7cb793150beb8: No bootstrap required, opened a new log
I20260929 18:37:22.773734 15336 ts_tablet_manager.cc:1397] T cbccb7dca15c4bdfa1618d176011a9aa P fdfd0eb5ab40468594d7cb793150beb8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 18:37:22.774484 15336 raft_consensus.cc:348] T cbccb7dca15c4bdfa1618d176011a9aa P fdfd0eb5ab40468594d7cb793150beb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdfd0eb5ab40468594d7cb793150beb8" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 45983 } } peers { permanent_uuid: "87b61322896e4e0c988d10faf801c9b5" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 32859 } } peers { permanent_uuid: "945f5bbc44574d57a45b266d384d2c59" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 32827 } }
I20260929 18:37:22.774616 15336 raft_consensus.cc:374] T cbccb7dca15c4bdfa1618d176011a9aa P fdfd0eb5ab40468594d7cb793150beb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:37:22.774647 15335 tablet_bootstrap.cc:654] T cbccb7dca15c4bdfa1618d176011a9aa P 945f5bbc44574d57a45b266d384d2c59: Neither blocks nor log segments found. Creating new log.
I20260929 18:37:22.774660 15336 raft_consensus.cc:729] T cbccb7dca15c4bdfa1618d176011a9aa P fdfd0eb5ab40468594d7cb793150beb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdfd0eb5ab40468594d7cb793150beb8, State: Initialized, Role: FOLLOWER
I20260929 18:37:22.774776 15336 consensus_queue.cc:260] T cbccb7dca15c4bdfa1618d176011a9aa P fdfd0eb5ab40468594d7cb793150beb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdfd0eb5ab40468594d7cb793150beb8" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 45983 } } peers { permanent_uuid: "87b61322896e4e0c988d10faf801c9b5" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 32859 } } peers { permanent_uuid: "945f5bbc44574d57a45b266d384d2c59" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 32827 } }
I20260929 18:37:22.775059 15336 ts_tablet_manager.cc:1428] T cbccb7dca15c4bdfa1618d176011a9aa P fdfd0eb5ab40468594d7cb793150beb8: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260929 18:37:22.775825 15360 ts_tablet_manager.cc:1939] T 6def05ee84cb41aeb511d2517914e9d3 P fdfd0eb5ab40468594d7cb793150beb8: Deleting consensus metadata
I20260929 18:37:22.775972 15335 tablet_bootstrap.cc:492] T cbccb7dca15c4bdfa1618d176011a9aa P 945f5bbc44574d57a45b266d384d2c59: No bootstrap required, opened a new log
I20260929 18:37:22.776041 15335 ts_tablet_manager.cc:1397] T cbccb7dca15c4bdfa1618d176011a9aa P 945f5bbc44574d57a45b266d384d2c59: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260929 18:37:22.776480 15335 raft_consensus.cc:348] T cbccb7dca15c4bdfa1618d176011a9aa P 945f5bbc44574d57a45b266d384d2c59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdfd0eb5ab40468594d7cb793150beb8" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 45983 } } peers { permanent_uuid: "87b61322896e4e0c988d10faf801c9b5" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 32859 } } peers { permanent_uuid: "945f5bbc44574d57a45b266d384d2c59" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 32827 } }
I20260929 18:37:22.776576 15335 raft_consensus.cc:374] T cbccb7dca15c4bdfa1618d176011a9aa P 945f5bbc44574d57a45b266d384d2c59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:37:22.776610 15335 raft_consensus.cc:729] T cbccb7dca15c4bdfa1618d176011a9aa P 945f5bbc44574d57a45b266d384d2c59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 945f5bbc44574d57a45b266d384d2c59, State: Initialized, Role: FOLLOWER
I20260929 18:37:22.776717 15335 consensus_queue.cc:260] T cbccb7dca15c4bdfa1618d176011a9aa P 945f5bbc44574d57a45b266d384d2c59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdfd0eb5ab40468594d7cb793150beb8" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 45983 } } peers { permanent_uuid: "87b61322896e4e0c988d10faf801c9b5" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 32859 } } peers { permanent_uuid: "945f5bbc44574d57a45b266d384d2c59" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 32827 } }
I20260929 18:37:22.776957 15335 ts_tablet_manager.cc:1428] T cbccb7dca15c4bdfa1618d176011a9aa P 945f5bbc44574d57a45b266d384d2c59: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260929 18:37:22.777360 14814 catalog_manager.cc:4841] TS fdfd0eb5ab40468594d7cb793150beb8 (127.13.100.1:45983): tablet 6def05ee84cb41aeb511d2517914e9d3 (table test-table [id=ada2a9ea621d4604a9c7897fa7c89990]) successfully deleted
I20260929 18:37:22.771492 15337 tablet_bootstrap.cc:654] T cbccb7dca15c4bdfa1618d176011a9aa P 87b61322896e4e0c988d10faf801c9b5: Neither blocks nor log segments found. Creating new log.
I20260929 18:37:22.778399 15359 ts_tablet_manager.cc:1939] T 6def05ee84cb41aeb511d2517914e9d3 P 87b61322896e4e0c988d10faf801c9b5: Deleting consensus metadata
I20260929 18:37:22.779186 15358 raft_consensus.cc:2229] T 6def05ee84cb41aeb511d2517914e9d3 P 945f5bbc44574d57a45b266d384d2c59 [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:37:22.779536 14815 catalog_manager.cc:4841] TS 87b61322896e4e0c988d10faf801c9b5 (127.13.100.2:32859): tablet 6def05ee84cb41aeb511d2517914e9d3 (table test-table [id=ada2a9ea621d4604a9c7897fa7c89990]) successfully deleted
I20260929 18:37:22.779587 15358 raft_consensus.cc:2258] T 6def05ee84cb41aeb511d2517914e9d3 P 945f5bbc44574d57a45b266d384d2c59 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:37:22.780369 15337 tablet_bootstrap.cc:492] T cbccb7dca15c4bdfa1618d176011a9aa P 87b61322896e4e0c988d10faf801c9b5: No bootstrap required, opened a new log
I20260929 18:37:22.780454 15337 ts_tablet_manager.cc:1397] T cbccb7dca15c4bdfa1618d176011a9aa P 87b61322896e4e0c988d10faf801c9b5: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20260929 18:37:22.780864 15337 raft_consensus.cc:348] T cbccb7dca15c4bdfa1618d176011a9aa P 87b61322896e4e0c988d10faf801c9b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdfd0eb5ab40468594d7cb793150beb8" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 45983 } } peers { permanent_uuid: "87b61322896e4e0c988d10faf801c9b5" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 32859 } } peers { permanent_uuid: "945f5bbc44574d57a45b266d384d2c59" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 32827 } }
I20260929 18:37:22.780967 15337 raft_consensus.cc:374] T cbccb7dca15c4bdfa1618d176011a9aa P 87b61322896e4e0c988d10faf801c9b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:37:22.780997 15337 raft_consensus.cc:729] T cbccb7dca15c4bdfa1618d176011a9aa P 87b61322896e4e0c988d10faf801c9b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87b61322896e4e0c988d10faf801c9b5, State: Initialized, Role: FOLLOWER
I20260929 18:37:22.781049 15358 ts_tablet_manager.cc:1905] T 6def05ee84cb41aeb511d2517914e9d3 P 945f5bbc44574d57a45b266d384d2c59: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:37:22.781107 15337 consensus_queue.cc:260] T cbccb7dca15c4bdfa1618d176011a9aa P 87b61322896e4e0c988d10faf801c9b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdfd0eb5ab40468594d7cb793150beb8" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 45983 } } peers { permanent_uuid: "87b61322896e4e0c988d10faf801c9b5" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 32859 } } peers { permanent_uuid: "945f5bbc44574d57a45b266d384d2c59" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 32827 } }
I20260929 18:37:22.781352 15337 ts_tablet_manager.cc:1428] T cbccb7dca15c4bdfa1618d176011a9aa P 87b61322896e4e0c988d10faf801c9b5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 18:37:22.783869 15358 ts_tablet_manager.cc:1918] T 6def05ee84cb41aeb511d2517914e9d3 P 945f5bbc44574d57a45b266d384d2c59: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:37:22.783980 15358 log.cc:1196] T 6def05ee84cb41aeb511d2517914e9d3 P 945f5bbc44574d57a45b266d384d2c59: Deleting WAL directory at /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707036041078-13712-0/minicluster-data/ts-2/wal/wals/6def05ee84cb41aeb511d2517914e9d3
I20260929 18:37:22.784256 15358 ts_tablet_manager.cc:1939] T 6def05ee84cb41aeb511d2517914e9d3 P 945f5bbc44574d57a45b266d384d2c59: Deleting consensus metadata
I20260929 18:37:22.785195 14814 catalog_manager.cc:4841] TS 945f5bbc44574d57a45b266d384d2c59 (127.13.100.3:32827): tablet 6def05ee84cb41aeb511d2517914e9d3 (table test-table [id=ada2a9ea621d4604a9c7897fa7c89990]) successfully deleted
W20260929 18:37:22.857260 15313 tablet.cc:2369] T cbccb7dca15c4bdfa1618d176011a9aa P 945f5bbc44574d57a45b266d384d2c59: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 18:37:22.889475 15173 tablet.cc:2369] T cbccb7dca15c4bdfa1618d176011a9aa P 87b61322896e4e0c988d10faf801c9b5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 18:37:22.922241 15033 tablet.cc:2369] T cbccb7dca15c4bdfa1618d176011a9aa P fdfd0eb5ab40468594d7cb793150beb8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:37:22.930799 15343 raft_consensus.cc:482] T cbccb7dca15c4bdfa1618d176011a9aa P fdfd0eb5ab40468594d7cb793150beb8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:37:22.930955 15343 raft_consensus.cc:504] T cbccb7dca15c4bdfa1618d176011a9aa P fdfd0eb5ab40468594d7cb793150beb8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdfd0eb5ab40468594d7cb793150beb8" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 45983 } } peers { permanent_uuid: "87b61322896e4e0c988d10faf801c9b5" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 32859 } } peers { permanent_uuid: "945f5bbc44574d57a45b266d384d2c59" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 32827 } }
I20260929 18:37:22.932085 15343 leader_election.cc:288] T cbccb7dca15c4bdfa1618d176011a9aa P fdfd0eb5ab40468594d7cb793150beb8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87b61322896e4e0c988d10faf801c9b5 (127.13.100.2:32859), 945f5bbc44574d57a45b266d384d2c59 (127.13.100.3:32827)
I20260929 18:37:22.935784 15331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:22.932414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.1:39943 (local address 127.13.100.2:32859)
0929 18:37:22.932565 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:22.932571 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:22.932666 (+ 95us) server_negotiation.cc:407] Connection header received
0929 18:37:22.932884 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:22.932888 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:22.932939 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:22.933010 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:37:22.933289 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:22.933794 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:22.934829 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:22.935213 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:22.935349 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:22.935405 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:22.935599 (+ 194us) server_negotiation.cc:299] Negotiation successful
0929 18:37:22.935664 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 18:37:22.936028 15333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:22.932573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.1:44373 (local address 127.13.100.3:32827)
0929 18:37:22.932696 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:22.932700 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:22.932933 (+ 233us) server_negotiation.cc:407] Connection header received
0929 18:37:22.933026 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:22.933029 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:22.933076 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:22.933160 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:37:22.933418 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:22.933970 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:22.935020 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:22.935470 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:22.935589 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:22.935638 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:22.935849 (+ 211us) server_negotiation.cc:299] Negotiation successful
0929 18:37:22.935913 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 18:37:22.936040 15365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:22.932434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.3:32827 (local address 127.13.100.1:44373)
0929 18:37:22.932825 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:22.932840 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:22.932940 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:22.933161 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:22.933164 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:22.933180 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:37:22.933310 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:22.933315 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:22.934071 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:22.934077 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:22.934900 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:22.934908 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:22.935580 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:22.935583 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:22.935695 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:22.935740 (+ 45us) client_negotiation.cc:769] Sending connection context
0929 18:37:22.935834 (+ 94us) client_negotiation.cc:240] Negotiation successful
0929 18:37:22.935896 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":116,"threads_started":1}
I20260929 18:37:22.936044 15364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:22.932283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.2:32859 (local address 127.13.100.1:39943)
0929 18:37:22.932560 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:22.932575 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:22.932755 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:22.933053 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:22.933056 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:22.933069 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:37:22.933184 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:22.933190 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:22.933921 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:22.933924 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:22.934708 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:22.934716 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:22.935349 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:22.935373 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:22.935480 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:22.935497 (+ 17us) client_negotiation.cc:769] Sending connection context
0929 18:37:22.935585 (+ 88us) client_negotiation.cc:240] Negotiation successful
0929 18:37:22.935649 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":75,"threads_started":1}
I20260929 18:37:22.936691 15113 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbccb7dca15c4bdfa1618d176011a9aa" candidate_uuid: "fdfd0eb5ab40468594d7cb793150beb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87b61322896e4e0c988d10faf801c9b5" is_pre_election: true
I20260929 18:37:22.936674 15266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbccb7dca15c4bdfa1618d176011a9aa" candidate_uuid: "fdfd0eb5ab40468594d7cb793150beb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "945f5bbc44574d57a45b266d384d2c59" is_pre_election: true
I20260929 18:37:22.936834 15113 raft_consensus.cc:2454] T cbccb7dca15c4bdfa1618d176011a9aa P 87b61322896e4e0c988d10faf801c9b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdfd0eb5ab40468594d7cb793150beb8 in term 0.
I20260929 18:37:22.936903 15266 raft_consensus.cc:2454] T cbccb7dca15c4bdfa1618d176011a9aa P 945f5bbc44574d57a45b266d384d2c59 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdfd0eb5ab40468594d7cb793150beb8 in term 0.
I20260929 18:37:22.937191 14919 leader_election.cc:302] T cbccb7dca15c4bdfa1618d176011a9aa P fdfd0eb5ab40468594d7cb793150beb8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87b61322896e4e0c988d10faf801c9b5, fdfd0eb5ab40468594d7cb793150beb8; no voters:
I20260929 18:37:22.937470 15343 raft_consensus.cc:2781] T cbccb7dca15c4bdfa1618d176011a9aa P fdfd0eb5ab40468594d7cb793150beb8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:37:22.937548 15343 raft_consensus.cc:482] T cbccb7dca15c4bdfa1618d176011a9aa P fdfd0eb5ab40468594d7cb793150beb8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:37:22.937601 15343 raft_consensus.cc:3037] T cbccb7dca15c4bdfa1618d176011a9aa P fdfd0eb5ab40468594d7cb793150beb8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:37:22.938604 15343 raft_consensus.cc:504] T cbccb7dca15c4bdfa1618d176011a9aa P fdfd0eb5ab40468594d7cb793150beb8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdfd0eb5ab40468594d7cb793150beb8" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 45983 } } peers { permanent_uuid: "87b61322896e4e0c988d10faf801c9b5" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 32859 } } peers { permanent_uuid: "945f5bbc44574d57a45b266d384d2c59" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 32827 } }
I20260929 18:37:22.938959 15343 leader_election.cc:288] T cbccb7dca15c4bdfa1618d176011a9aa P fdfd0eb5ab40468594d7cb793150beb8 [CANDIDATE]: Term 1 election: Requested vote from peers 87b61322896e4e0c988d10faf801c9b5 (127.13.100.2:32859), 945f5bbc44574d57a45b266d384d2c59 (127.13.100.3:32827)
I20260929 18:37:22.939296 15113 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbccb7dca15c4bdfa1618d176011a9aa" candidate_uuid: "fdfd0eb5ab40468594d7cb793150beb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87b61322896e4e0c988d10faf801c9b5"
I20260929 18:37:22.939435 15113 raft_consensus.cc:3037] T cbccb7dca15c4bdfa1618d176011a9aa P 87b61322896e4e0c988d10faf801c9b5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:37:22.940372 15113 raft_consensus.cc:2454] T cbccb7dca15c4bdfa1618d176011a9aa P 87b61322896e4e0c988d10faf801c9b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdfd0eb5ab40468594d7cb793150beb8 in term 1.
I20260929 18:37:22.940629 15266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbccb7dca15c4bdfa1618d176011a9aa" candidate_uuid: "fdfd0eb5ab40468594d7cb793150beb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "945f5bbc44574d57a45b266d384d2c59"
I20260929 18:37:22.940709 14919 leader_election.cc:302] T cbccb7dca15c4bdfa1618d176011a9aa P fdfd0eb5ab40468594d7cb793150beb8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87b61322896e4e0c988d10faf801c9b5, fdfd0eb5ab40468594d7cb793150beb8; no voters:
I20260929 18:37:22.940752 15266 raft_consensus.cc:3037] T cbccb7dca15c4bdfa1618d176011a9aa P 945f5bbc44574d57a45b266d384d2c59 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:37:22.940888 15343 raft_consensus.cc:2781] T cbccb7dca15c4bdfa1618d176011a9aa P fdfd0eb5ab40468594d7cb793150beb8 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:37:22.941133 15343 raft_consensus.cc:686] T cbccb7dca15c4bdfa1618d176011a9aa P fdfd0eb5ab40468594d7cb793150beb8 [term 1 LEADER]: Becoming Leader. State: Replica: fdfd0eb5ab40468594d7cb793150beb8, State: Running, Role: LEADER
I20260929 18:37:22.941401 15343 consensus_queue.cc:237] T cbccb7dca15c4bdfa1618d176011a9aa P fdfd0eb5ab40468594d7cb793150beb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdfd0eb5ab40468594d7cb793150beb8" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 45983 } } peers { permanent_uuid: "87b61322896e4e0c988d10faf801c9b5" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 32859 } } peers { permanent_uuid: "945f5bbc44574d57a45b266d384d2c59" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 32827 } }
I20260929 18:37:22.941785 15266 raft_consensus.cc:2454] T cbccb7dca15c4bdfa1618d176011a9aa P 945f5bbc44574d57a45b266d384d2c59 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdfd0eb5ab40468594d7cb793150beb8 in term 1.
I20260929 18:37:22.943766 14827 catalog_manager.cc:5495] T cbccb7dca15c4bdfa1618d176011a9aa P fdfd0eb5ab40468594d7cb793150beb8 reported cstate change: term changed from 0 to 1, leader changed from <none> to fdfd0eb5ab40468594d7cb793150beb8 (127.13.100.1). New cstate: current_term: 1 leader_uuid: "fdfd0eb5ab40468594d7cb793150beb8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdfd0eb5ab40468594d7cb793150beb8" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 45983 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "87b61322896e4e0c988d10faf801c9b5" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 32859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "945f5bbc44574d57a45b266d384d2c59" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 32827 } health_report { overall_health: UNKNOWN } } }
Sep 29 18:37:23 dist-test-slave-j1xs krb5kdc[14784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707041, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.62@KRBTEST.COM for kudu/127.13.100.62@KRBTEST.COM
I20260929 18:37:23.008168 15373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:22.999627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:34301 (local address 127.0.0.1:48616)
0929 18:37:22.999995 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:23.000008 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:23.000110 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:23.000349 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:23.000352 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:23.000825 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:37:23.000915 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:23.000921 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:23.001712 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:23.001715 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:23.002345 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:23.002354 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:23.002841 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:23.002843 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:23.002902 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:23.003525 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:37:23.003537 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:37:23.005103 (+ 1566us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:37:23.006791 (+ 1688us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:23.006798 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:23.006802 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:23.007016 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:23.007276 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:23.007278 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:23.007279 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:23.007352 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:23.007669 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:37:23.007674 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:37:23.007769 (+ 95us) client_negotiation.cc:769] Sending connection context
0929 18:37:23.007869 (+ 100us) client_negotiation.cc:240] Negotiation successful
0929 18:37:23.007982 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":159,"threads_started":1}
I20260929 18:37:23.008185 14878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:22.999737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48616 (local address 127.13.100.62:34301)
0929 18:37:22.999896 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:22.999901 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:23.000100 (+ 199us) server_negotiation.cc:407] Connection header received
0929 18:37:23.000192 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:23.000194 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:23.000234 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:23.000308 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:23.001020 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:23.001584 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:23.002515 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:23.002732 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:23.002827 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:23.005258 (+ 2431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:23.005261 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:23.005263 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:37:23.005285 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:23.006694 (+ 1409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:23.007126 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:23.007129 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:23.007130 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:23.007196 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:23.007426 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:23.007428 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:23.007429 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:23.007583 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:23.007656 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:23.007921 (+ 265us) server_negotiation.cc:299] Negotiation successful
0929 18:37:23.008034 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":75}
I20260929 18:37:23.012143 14827 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.13.100.62@KRBTEST.COM'} at 127.0.0.1:48616:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 18:37:23.012877 14827 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:37:23.017424 15106 tablet_service.cc:1467] Processing CreateTablet for tablet a1976b6591094719b6ce4a73186da2ef (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9fdd2f2a5cd04c9dbd114d64860c5b56]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 18:37:23.017426 14966 tablet_service.cc:1467] Processing CreateTablet for tablet a1976b6591094719b6ce4a73186da2ef (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9fdd2f2a5cd04c9dbd114d64860c5b56]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 18:37:23.017621 15246 tablet_service.cc:1467] Processing CreateTablet for tablet a1976b6591094719b6ce4a73186da2ef (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9fdd2f2a5cd04c9dbd114d64860c5b56]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 18:37:23.017799 15106 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1976b6591094719b6ce4a73186da2ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:37:23.017812 14966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1976b6591094719b6ce4a73186da2ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:37:23.018103 15246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1976b6591094719b6ce4a73186da2ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:37:23.020696 15335 tablet_bootstrap.cc:492] T a1976b6591094719b6ce4a73186da2ef P 945f5bbc44574d57a45b266d384d2c59: Bootstrap starting.
I20260929 18:37:23.020694 15336 tablet_bootstrap.cc:492] T a1976b6591094719b6ce4a73186da2ef P fdfd0eb5ab40468594d7cb793150beb8: Bootstrap starting.
I20260929 18:37:23.021447 15335 tablet_bootstrap.cc:654] T a1976b6591094719b6ce4a73186da2ef P 945f5bbc44574d57a45b266d384d2c59: Neither blocks nor log segments found. Creating new log.
I20260929 18:37:23.021575 15336 tablet_bootstrap.cc:654] T a1976b6591094719b6ce4a73186da2ef P fdfd0eb5ab40468594d7cb793150beb8: Neither blocks nor log segments found. Creating new log.
I20260929 18:37:23.022274 15337 tablet_bootstrap.cc:492] T a1976b6591094719b6ce4a73186da2ef P 87b61322896e4e0c988d10faf801c9b5: Bootstrap starting.
I20260929 18:37:23.022522 15335 tablet_bootstrap.cc:492] T a1976b6591094719b6ce4a73186da2ef P 945f5bbc44574d57a45b266d384d2c59: No bootstrap required, opened a new log
I20260929 18:37:23.022599 15335 ts_tablet_manager.cc:1397] T a1976b6591094719b6ce4a73186da2ef P 945f5bbc44574d57a45b266d384d2c59: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260929 18:37:23.023029 15335 raft_consensus.cc:348] T a1976b6591094719b6ce4a73186da2ef P 945f5bbc44574d57a45b266d384d2c59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdfd0eb5ab40468594d7cb793150beb8" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 45983 } } peers { permanent_uuid: "87b61322896e4e0c988d10faf801c9b5" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 32859 } } peers { permanent_uuid: "945f5bbc44574d57a45b266d384d2c59" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 32827 } }
I20260929 18:37:23.023165 15335 raft_consensus.cc:374] T a1976b6591094719b6ce4a73186da2ef P 945f5bbc44574d57a45b266d384d2c59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:37:23.023211 15335 raft_consensus.cc:729] T a1976b6591094719b6ce4a73186da2ef P 945f5bbc44574d57a45b266d384d2c59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 945f5bbc44574d57a45b266d384d2c59, State: Initialized, Role: FOLLOWER
I20260929 18:37:23.023321 15337 tablet_bootstrap.cc:654] T a1976b6591094719b6ce4a73186da2ef P 87b61322896e4e0c988d10faf801c9b5: Neither blocks nor log segments found. Creating new log.
I20260929 18:37:23.023365 15335 consensus_queue.cc:260] T a1976b6591094719b6ce4a73186da2ef P 945f5bbc44574d57a45b266d384d2c59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdfd0eb5ab40468594d7cb793150beb8" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 45983 } } peers { permanent_uuid: "87b61322896e4e0c988d10faf801c9b5" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 32859 } } peers { permanent_uuid: "945f5bbc44574d57a45b266d384d2c59" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 32827 } }
I20260929 18:37:23.023696 15335 ts_tablet_manager.cc:1428] T a1976b6591094719b6ce4a73186da2ef P 945f5bbc44574d57a45b266d384d2c59: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260929 18:37:23.023643 15354 tablet_replica.cc:439] T a1976b6591094719b6ce4a73186da2ef P 945f5bbc44574d57a45b266d384d2c59: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdfd0eb5ab40468594d7cb793150beb8" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 45983 } } peers { permanent_uuid: "87b61322896e4e0c988d10faf801c9b5" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 32859 } } peers { permanent_uuid: "945f5bbc44574d57a45b266d384d2c59" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 32827 } } }
I20260929 18:37:23.023777 15354 tablet_replica.cc:442] T a1976b6591094719b6ce4a73186da2ef P 945f5bbc44574d57a45b266d384d2c59: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 18:37:23.024350 15336 tablet_bootstrap.cc:492] T a1976b6591094719b6ce4a73186da2ef P fdfd0eb5ab40468594d7cb793150beb8: No bootstrap required, opened a new log
I20260929 18:37:23.024458 15336 ts_tablet_manager.cc:1397] T a1976b6591094719b6ce4a73186da2ef P fdfd0eb5ab40468594d7cb793150beb8: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20260929 18:37:23.024863 15336 raft_consensus.cc:348] T a1976b6591094719b6ce4a73186da2ef P fdfd0eb5ab40468594d7cb793150beb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdfd0eb5ab40468594d7cb793150beb8" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 45983 } } peers { permanent_uuid: "87b61322896e4e0c988d10faf801c9b5" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 32859 } } peers { permanent_uuid: "945f5bbc44574d57a45b266d384d2c59" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 32827 } }
I20260929 18:37:23.024976 15336 raft_consensus.cc:374] T a1976b6591094719b6ce4a73186da2ef P fdfd0eb5ab40468594d7cb793150beb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:37:23.025027 15336 raft_consensus.cc:729] T a1976b6591094719b6ce4a73186da2ef P fdfd0eb5ab40468594d7cb793150beb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdfd0eb5ab40468594d7cb793150beb8, State: Initialized, Role: FOLLOWER
I20260929 18:37:23.025166 15336 consensus_queue.cc:260] T a1976b6591094719b6ce4a73186da2ef P fdfd0eb5ab40468594d7cb793150beb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdfd0eb5ab40468594d7cb793150beb8" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 45983 } } peers { permanent_uuid: "87b61322896e4e0c988d10faf801c9b5" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 32859 } } peers { permanent_uuid: "945f5bbc44574d57a45b266d384d2c59" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 32827 } }
I20260929 18:37:23.025456 15336 ts_tablet_manager.cc:1428] T a1976b6591094719b6ce4a73186da2ef P fdfd0eb5ab40468594d7cb793150beb8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260929 18:37:23.025408 15343 tablet_replica.cc:439] T a1976b6591094719b6ce4a73186da2ef P fdfd0eb5ab40468594d7cb793150beb8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdfd0eb5ab40468594d7cb793150beb8" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 45983 } } peers { permanent_uuid: "87b61322896e4e0c988d10faf801c9b5" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 32859 } } peers { permanent_uuid: "945f5bbc44574d57a45b266d384d2c59" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 32827 } } }
I20260929 18:37:23.025660 15343 tablet_replica.cc:442] T a1976b6591094719b6ce4a73186da2ef P fdfd0eb5ab40468594d7cb793150beb8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 18:37:23.026428 15337 tablet_bootstrap.cc:492] T a1976b6591094719b6ce4a73186da2ef P 87b61322896e4e0c988d10faf801c9b5: No bootstrap required, opened a new log
I20260929 18:37:23.026533 15337 ts_tablet_manager.cc:1397] T a1976b6591094719b6ce4a73186da2ef P 87b61322896e4e0c988d10faf801c9b5: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260929 18:37:23.027034 15337 raft_consensus.cc:348] T a1976b6591094719b6ce4a73186da2ef P 87b61322896e4e0c988d10faf801c9b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdfd0eb5ab40468594d7cb793150beb8" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 45983 } } peers { permanent_uuid: "87b61322896e4e0c988d10faf801c9b5" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 32859 } } peers { permanent_uuid: "945f5bbc44574d57a45b266d384d2c59" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 32827 } }
I20260929 18:37:23.027143 15337 raft_consensus.cc:374] T a1976b6591094719b6ce4a73186da2ef P 87b61322896e4e0c988d10faf801c9b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:37:23.027168 15337 raft_consensus.cc:729] T a1976b6591094719b6ce4a73186da2ef P 87b61322896e4e0c988d10faf801c9b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87b61322896e4e0c988d10faf801c9b5, State: Initialized, Role: FOLLOWER
I20260929 18:37:23.027324 15337 consensus_queue.cc:260] T a1976b6591094719b6ce4a73186da2ef P 87b61322896e4e0c988d10faf801c9b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdfd0eb5ab40468594d7cb793150beb8" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 45983 } } peers { permanent_uuid: "87b61322896e4e0c988d10faf801c9b5" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 32859 } } peers { permanent_uuid: "945f5bbc44574d57a45b266d384d2c59" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 32827 } }
I20260929 18:37:23.027580 15342 tablet_replica.cc:439] T a1976b6591094719b6ce4a73186da2ef P 87b61322896e4e0c988d10faf801c9b5: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdfd0eb5ab40468594d7cb793150beb8" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 45983 } } peers { permanent_uuid: "87b61322896e4e0c988d10faf801c9b5" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 32859 } } peers { permanent_uuid: "945f5bbc44574d57a45b266d384d2c59" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 32827 } } }
I20260929 18:37:23.027647 15337 ts_tablet_manager.cc:1428] T a1976b6591094719b6ce4a73186da2ef P 87b61322896e4e0c988d10faf801c9b5: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260929 18:37:23.027719 15342 tablet_replica.cc:442] T a1976b6591094719b6ce4a73186da2ef P 87b61322896e4e0c988d10faf801c9b5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 18:37:23.231194 15342 raft_consensus.cc:482] T a1976b6591094719b6ce4a73186da2ef P 87b61322896e4e0c988d10faf801c9b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:37:23.231360 15342 raft_consensus.cc:504] T a1976b6591094719b6ce4a73186da2ef P 87b61322896e4e0c988d10faf801c9b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdfd0eb5ab40468594d7cb793150beb8" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 45983 } } peers { permanent_uuid: "87b61322896e4e0c988d10faf801c9b5" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 32859 } } peers { permanent_uuid: "945f5bbc44574d57a45b266d384d2c59" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 32827 } }
I20260929 18:37:23.232504 15342 leader_election.cc:288] T a1976b6591094719b6ce4a73186da2ef P 87b61322896e4e0c988d10faf801c9b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fdfd0eb5ab40468594d7cb793150beb8 (127.13.100.1:45983), 945f5bbc44574d57a45b266d384d2c59 (127.13.100.3:32827)
I20260929 18:37:23.237797 15378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:23.232849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.2:59535 (local address 127.13.100.1:45983)
0929 18:37:23.233106 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:23.233112 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:23.233131 (+ 19us) server_negotiation.cc:407] Connection header received
0929 18:37:23.233226 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:23.233230 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:23.233283 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:23.233392 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:37:23.233704 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:23.234587 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:23.236410 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:23.237050 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:23.237215 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:23.237260 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:23.237579 (+ 319us) server_negotiation.cc:299] Negotiation successful
0929 18:37:23.237655 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":86,"threads_started":1}
I20260929 18:37:23.237792 15333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:23.233246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.2:46207 (local address 127.13.100.3:32827)
0929 18:37:23.233388 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:23.233392 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:23.233444 (+ 52us) server_negotiation.cc:407] Connection header received
0929 18:37:23.233567 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:23.233569 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:23.233606 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:23.233713 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:37:23.234008 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:23.234730 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:23.236410 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:23.237050 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:23.237182 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:23.237247 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:23.237579 (+ 332us) server_negotiation.cc:299] Negotiation successful
0929 18:37:23.237655 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 18:37:23.237864 15379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:23.233127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.3:32827 (local address 127.13.100.2:46207)
0929 18:37:23.233375 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:23.233391 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:23.233470 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:23.233739 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:23.233741 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:23.233755 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:37:23.233840 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:23.233844 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:23.234837 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:23.234840 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:23.236252 (+ 1412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:23.236263 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:23.237223 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:23.237227 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:23.237417 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:23.237463 (+ 46us) client_negotiation.cc:769] Sending connection context
0929 18:37:23.237550 (+ 87us) client_negotiation.cc:240] Negotiation successful
0929 18:37:23.237666 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"mutex_wait_us":207,"thread_start_us":60,"threads_started":1}
I20260929 18:37:23.237864 15377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:23.232745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.1:45983 (local address 127.13.100.2:59535)
0929 18:37:23.232995 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:23.233011 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:23.233111 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:23.233421 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:23.233427 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:23.233456 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:37:23.233598 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:23.233603 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:23.234751 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:23.234754 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:23.236253 (+ 1499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:23.236263 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:23.237223 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:23.237227 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:23.237421 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:23.237464 (+ 43us) client_negotiation.cc:769] Sending connection context
0929 18:37:23.237550 (+ 86us) client_negotiation.cc:240] Negotiation successful
0929 18:37:23.237667 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"mutex_wait_us":113,"thread_start_us":97,"threads_started":1}
I20260929 18:37:23.238448 14986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1976b6591094719b6ce4a73186da2ef" candidate_uuid: "87b61322896e4e0c988d10faf801c9b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdfd0eb5ab40468594d7cb793150beb8" is_pre_election: true
I20260929 18:37:23.238448 15266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1976b6591094719b6ce4a73186da2ef" candidate_uuid: "87b61322896e4e0c988d10faf801c9b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "945f5bbc44574d57a45b266d384d2c59" is_pre_election: true
I20260929 18:37:23.238601 14986 raft_consensus.cc:2454] T a1976b6591094719b6ce4a73186da2ef P fdfd0eb5ab40468594d7cb793150beb8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87b61322896e4e0c988d10faf801c9b5 in term 0.
I20260929 18:37:23.238601 15266 raft_consensus.cc:2454] T a1976b6591094719b6ce4a73186da2ef P 945f5bbc44574d57a45b266d384d2c59 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87b61322896e4e0c988d10faf801c9b5 in term 0.
I20260929 18:37:23.238940 15058 leader_election.cc:302] T a1976b6591094719b6ce4a73186da2ef P 87b61322896e4e0c988d10faf801c9b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87b61322896e4e0c988d10faf801c9b5, fdfd0eb5ab40468594d7cb793150beb8; no voters:
I20260929 18:37:23.239174 15342 raft_consensus.cc:2781] T a1976b6591094719b6ce4a73186da2ef P 87b61322896e4e0c988d10faf801c9b5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:37:23.239259 15342 raft_consensus.cc:482] T a1976b6591094719b6ce4a73186da2ef P 87b61322896e4e0c988d10faf801c9b5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:37:23.239298 15342 raft_consensus.cc:3037] T a1976b6591094719b6ce4a73186da2ef P 87b61322896e4e0c988d10faf801c9b5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:37:23.240288 15342 raft_consensus.cc:504] T a1976b6591094719b6ce4a73186da2ef P 87b61322896e4e0c988d10faf801c9b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdfd0eb5ab40468594d7cb793150beb8" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 45983 } } peers { permanent_uuid: "87b61322896e4e0c988d10faf801c9b5" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 32859 } } peers { permanent_uuid: "945f5bbc44574d57a45b266d384d2c59" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 32827 } }
I20260929 18:37:23.240633 15342 leader_election.cc:288] T a1976b6591094719b6ce4a73186da2ef P 87b61322896e4e0c988d10faf801c9b5 [CANDIDATE]: Term 1 election: Requested vote from peers fdfd0eb5ab40468594d7cb793150beb8 (127.13.100.1:45983), 945f5bbc44574d57a45b266d384d2c59 (127.13.100.3:32827)
I20260929 18:37:23.241001 15266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1976b6591094719b6ce4a73186da2ef" candidate_uuid: "87b61322896e4e0c988d10faf801c9b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "945f5bbc44574d57a45b266d384d2c59"
I20260929 18:37:23.241001 14986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1976b6591094719b6ce4a73186da2ef" candidate_uuid: "87b61322896e4e0c988d10faf801c9b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdfd0eb5ab40468594d7cb793150beb8"
I20260929 18:37:23.241115 14986 raft_consensus.cc:3037] T a1976b6591094719b6ce4a73186da2ef P fdfd0eb5ab40468594d7cb793150beb8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:37:23.241115 15266 raft_consensus.cc:3037] T a1976b6591094719b6ce4a73186da2ef P 945f5bbc44574d57a45b266d384d2c59 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:37:23.242224 15266 raft_consensus.cc:2454] T a1976b6591094719b6ce4a73186da2ef P 945f5bbc44574d57a45b266d384d2c59 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87b61322896e4e0c988d10faf801c9b5 in term 1.
I20260929 18:37:23.242218 14986 raft_consensus.cc:2454] T a1976b6591094719b6ce4a73186da2ef P fdfd0eb5ab40468594d7cb793150beb8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87b61322896e4e0c988d10faf801c9b5 in term 1.
I20260929 18:37:23.242508 15058 leader_election.cc:302] T a1976b6591094719b6ce4a73186da2ef P 87b61322896e4e0c988d10faf801c9b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87b61322896e4e0c988d10faf801c9b5, fdfd0eb5ab40468594d7cb793150beb8; no voters:
I20260929 18:37:23.242667 15342 raft_consensus.cc:2781] T a1976b6591094719b6ce4a73186da2ef P 87b61322896e4e0c988d10faf801c9b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:37:23.242836 15342 raft_consensus.cc:686] T a1976b6591094719b6ce4a73186da2ef P 87b61322896e4e0c988d10faf801c9b5 [term 1 LEADER]: Becoming Leader. State: Replica: 87b61322896e4e0c988d10faf801c9b5, State: Running, Role: LEADER
I20260929 18:37:23.243098 15342 consensus_queue.cc:237] T a1976b6591094719b6ce4a73186da2ef P 87b61322896e4e0c988d10faf801c9b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdfd0eb5ab40468594d7cb793150beb8" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 45983 } } peers { permanent_uuid: "87b61322896e4e0c988d10faf801c9b5" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 32859 } } peers { permanent_uuid: "945f5bbc44574d57a45b266d384d2c59" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 32827 } }
I20260929 18:37:23.244361 15380 tablet_replica.cc:439] T a1976b6591094719b6ce4a73186da2ef P 87b61322896e4e0c988d10faf801c9b5: TxnStatusTablet state changed. Reason: New leader 87b61322896e4e0c988d10faf801c9b5. Latest consensus state: current_term: 1 leader_uuid: "87b61322896e4e0c988d10faf801c9b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdfd0eb5ab40468594d7cb793150beb8" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 45983 } } peers { permanent_uuid: "87b61322896e4e0c988d10faf801c9b5" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 32859 } } peers { permanent_uuid: "945f5bbc44574d57a45b266d384d2c59" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 32827 } } }
I20260929 18:37:23.244513 15380 tablet_replica.cc:442] T a1976b6591094719b6ce4a73186da2ef P 87b61322896e4e0c988d10faf801c9b5: This TxnStatusTablet replica's current role is: LEADER
I20260929 18:37:23.244791 15382 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 18:37:23.244941 15382 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 18:37:23.245744 14829 catalog_manager.cc:5495] T a1976b6591094719b6ce4a73186da2ef P 87b61322896e4e0c988d10faf801c9b5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 87b61322896e4e0c988d10faf801c9b5 (127.13.100.2). New cstate: current_term: 1 leader_uuid: "87b61322896e4e0c988d10faf801c9b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdfd0eb5ab40468594d7cb793150beb8" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 45983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "87b61322896e4e0c988d10faf801c9b5" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 32859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "945f5bbc44574d57a45b266d384d2c59" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 32827 } health_report { overall_health: UNKNOWN } } }
I20260929 18:37:23.263201 15373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:23.258814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.2:32859 (local address 127.0.0.1:40828)
0929 18:37:23.259106 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:23.259119 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:23.259214 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:23.259553 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:23.259556 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:23.259572 (+ 16us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 18:37:23.259672 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:23.259677 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:23.260692 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:23.260703 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:23.261353 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:23.261362 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:23.261982 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:23.261986 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:23.262059 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:23.262102 (+ 43us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 18:37:23.262901 (+ 799us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 18:37:23.262918 (+ 17us) client_negotiation.cc:769] Sending connection context
0929 18:37:23.262996 (+ 78us) client_negotiation.cc:240] Negotiation successful
0929 18:37:23.263082 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199}
I20260929 18:37:23.263214 15331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:23.258956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40828 (local address 127.13.100.2:32859)
0929 18:37:23.259090 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:23.259094 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:23.259178 (+ 84us) server_negotiation.cc:407] Connection header received
0929 18:37:23.259340 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:23.259343 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:23.259404 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:23.259511 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:37:23.259782 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:23.260557 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:23.261467 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:23.261811 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:23.261937 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:23.262242 (+ 305us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:37:23.262743 (+ 501us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:37:23.262882 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:23.263022 (+ 140us) server_negotiation.cc:299] Negotiation successful
0929 18:37:23.263076 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 18:37:23.266630 14986 raft_consensus.cc:1261] T a1976b6591094719b6ce4a73186da2ef P fdfd0eb5ab40468594d7cb793150beb8 [term 1 FOLLOWER]: Refusing update from remote peer 87b61322896e4e0c988d10faf801c9b5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:37:23.266664 15266 raft_consensus.cc:1261] T a1976b6591094719b6ce4a73186da2ef P 945f5bbc44574d57a45b266d384d2c59 [term 1 FOLLOWER]: Refusing update from remote peer 87b61322896e4e0c988d10faf801c9b5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:37:23.267143 15380 consensus_queue.cc:1035] T a1976b6591094719b6ce4a73186da2ef P 87b61322896e4e0c988d10faf801c9b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfd0eb5ab40468594d7cb793150beb8" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 45983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:37:23.267410 15342 consensus_queue.cc:1035] T a1976b6591094719b6ce4a73186da2ef P 87b61322896e4e0c988d10faf801c9b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "945f5bbc44574d57a45b266d384d2c59" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 32827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:37:23.268908 15343 tablet_replica.cc:439] T a1976b6591094719b6ce4a73186da2ef P fdfd0eb5ab40468594d7cb793150beb8: TxnStatusTablet state changed. Reason: New leader 87b61322896e4e0c988d10faf801c9b5. Latest consensus state: current_term: 1 leader_uuid: "87b61322896e4e0c988d10faf801c9b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdfd0eb5ab40468594d7cb793150beb8" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 45983 } } peers { permanent_uuid: "87b61322896e4e0c988d10faf801c9b5" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 32859 } } peers { permanent_uuid: "945f5bbc44574d57a45b266d384d2c59" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 32827 } } }
I20260929 18:37:23.269037 15343 tablet_replica.cc:442] T a1976b6591094719b6ce4a73186da2ef P fdfd0eb5ab40468594d7cb793150beb8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 18:37:23.269114 15354 tablet_replica.cc:439] T a1976b6591094719b6ce4a73186da2ef P 945f5bbc44574d57a45b266d384d2c59: TxnStatusTablet state changed. Reason: New leader 87b61322896e4e0c988d10faf801c9b5. Latest consensus state: current_term: 1 leader_uuid: "87b61322896e4e0c988d10faf801c9b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdfd0eb5ab40468594d7cb793150beb8" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 45983 } } peers { permanent_uuid: "87b61322896e4e0c988d10faf801c9b5" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 32859 } } peers { permanent_uuid: "945f5bbc44574d57a45b266d384d2c59" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 32827 } } }
I20260929 18:37:23.269217 15354 tablet_replica.cc:442] T a1976b6591094719b6ce4a73186da2ef P 945f5bbc44574d57a45b266d384d2c59: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 18:37:23.271251 15385 mvcc.cc:205] Tried to move back new op lower bound from 7334736049214005248 to 7334736049125912576. Current Snapshot: MvccSnapshot[applied={T|T < 7334736049214005248}]
I20260929 18:37:23.271361 15342 tablet_replica.cc:439] T a1976b6591094719b6ce4a73186da2ef P 87b61322896e4e0c988d10faf801c9b5: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "87b61322896e4e0c988d10faf801c9b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdfd0eb5ab40468594d7cb793150beb8" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 45983 } } peers { permanent_uuid: "87b61322896e4e0c988d10faf801c9b5" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 32859 } } peers { permanent_uuid: "945f5bbc44574d57a45b266d384d2c59" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 32827 } } }
I20260929 18:37:23.271454 15342 tablet_replica.cc:442] T a1976b6591094719b6ce4a73186da2ef P 87b61322896e4e0c988d10faf801c9b5: This TxnStatusTablet replica's current role is: LEADER
I20260929 18:37:23.271735 15343 tablet_replica.cc:439] T a1976b6591094719b6ce4a73186da2ef P fdfd0eb5ab40468594d7cb793150beb8: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "87b61322896e4e0c988d10faf801c9b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdfd0eb5ab40468594d7cb793150beb8" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 45983 } } peers { permanent_uuid: "87b61322896e4e0c988d10faf801c9b5" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 32859 } } peers { permanent_uuid: "945f5bbc44574d57a45b266d384d2c59" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 32827 } } }
I20260929 18:37:23.271853 15343 tablet_replica.cc:442] T a1976b6591094719b6ce4a73186da2ef P fdfd0eb5ab40468594d7cb793150beb8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 18:37:23.272712 15380 tablet_replica.cc:439] T a1976b6591094719b6ce4a73186da2ef P 87b61322896e4e0c988d10faf801c9b5: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "87b61322896e4e0c988d10faf801c9b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdfd0eb5ab40468594d7cb793150beb8" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 45983 } } peers { permanent_uuid: "87b61322896e4e0c988d10faf801c9b5" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 32859 } } peers { permanent_uuid: "945f5bbc44574d57a45b266d384d2c59" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 32827 } } }
I20260929 18:37:23.272814 15380 tablet_replica.cc:442] T a1976b6591094719b6ce4a73186da2ef P 87b61322896e4e0c988d10faf801c9b5: This TxnStatusTablet replica's current role is: LEADER
I20260929 18:37:23.274036 15387 mvcc.cc:205] Tried to move back new op lower bound from 7334736049214005248 to 7334736049125912576. Current Snapshot: MvccSnapshot[applied={T|T < 7334736049214005248}]
I20260929 18:37:23.274318 15354 tablet_replica.cc:439] T a1976b6591094719b6ce4a73186da2ef P 945f5bbc44574d57a45b266d384d2c59: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "87b61322896e4e0c988d10faf801c9b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdfd0eb5ab40468594d7cb793150beb8" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 45983 } } peers { permanent_uuid: "87b61322896e4e0c988d10faf801c9b5" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 32859 } } peers { permanent_uuid: "945f5bbc44574d57a45b266d384d2c59" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 32827 } } }
I20260929 18:37:23.274415 15354 tablet_replica.cc:442] T a1976b6591094719b6ce4a73186da2ef P 945f5bbc44574d57a45b266d384d2c59: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 18:37:23.284201 15378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:23.280269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53898 (local address 127.13.100.1:45983)
0929 18:37:23.280434 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:23.280437 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:23.280516 (+ 79us) server_negotiation.cc:407] Connection header received
0929 18:37:23.280669 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:23.280671 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:23.280719 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:23.280801 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:37:23.281043 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:23.281518 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:23.282739 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:23.282976 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:23.283057 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:23.283352 (+ 295us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:37:23.283791 (+ 439us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:37:23.283885 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:23.283984 (+ 99us) server_negotiation.cc:299] Negotiation successful
0929 18:37:23.284033 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260929 18:37:23.295292 15398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:23.291079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.2:32859 (local address 127.13.100.1:39729)
0929 18:37:23.291356 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:23.291368 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:23.291451 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:23.291931 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:23.291933 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:23.291941 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 18:37:23.292029 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:23.292033 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:23.292987 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:23.292990 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:23.293692 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:23.293701 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:23.294652 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:23.294655 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:23.294716 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:23.294756 (+ 40us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 18:37:23.295035 (+ 279us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 18:37:23.295039 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 18:37:23.295087 (+ 48us) client_negotiation.cc:240] Negotiation successful
0929 18:37:23.295170 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":96,"threads_started":1}
I20260929 18:37:23.295296 15331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:23.291226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.1:39729 (local address 127.13.100.2:32859)
0929 18:37:23.291358 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:23.291363 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:23.291442 (+ 79us) server_negotiation.cc:407] Connection header received
0929 18:37:23.291771 (+ 329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:23.291774 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:23.291827 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:23.291895 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:37:23.292125 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:23.292887 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:23.293803 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:23.294540 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:23.294651 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:23.294882 (+ 231us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:37:23.294957 (+ 75us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:37:23.295026 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:23.295117 (+ 91us) server_negotiation.cc:299] Negotiation successful
0929 18:37:23.295166 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 18:37:23.302872 15266 raft_consensus.cc:1261] T cbccb7dca15c4bdfa1618d176011a9aa P 945f5bbc44574d57a45b266d384d2c59 [term 1 FOLLOWER]: Refusing update from remote peer fdfd0eb5ab40468594d7cb793150beb8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:37:23.302942 15113 raft_consensus.cc:1261] T cbccb7dca15c4bdfa1618d176011a9aa P 87b61322896e4e0c988d10faf801c9b5 [term 1 FOLLOWER]: Refusing update from remote peer fdfd0eb5ab40468594d7cb793150beb8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:37:23.303347 15366 consensus_queue.cc:1035] T cbccb7dca15c4bdfa1618d176011a9aa P fdfd0eb5ab40468594d7cb793150beb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "945f5bbc44574d57a45b266d384d2c59" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 32827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:37:23.303566 15343 consensus_queue.cc:1035] T cbccb7dca15c4bdfa1618d176011a9aa P fdfd0eb5ab40468594d7cb793150beb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87b61322896e4e0c988d10faf801c9b5" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 32859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:37:23.330067 15402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:23.322521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.1:45983 (local address 127.13.100.2:39877)
0929 18:37:23.326141 (+ 3620us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:23.326157 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:23.326271 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:23.326554 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:23.326556 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:23.326564 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 18:37:23.326699 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:23.326705 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:23.327749 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:23.327753 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:23.328728 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:23.328738 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:23.329236 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:23.329240 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:23.329345 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:23.329397 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 18:37:23.329700 (+ 303us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 18:37:23.329706 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 18:37:23.329774 (+ 68us) client_negotiation.cc:240] Negotiation successful
0929 18:37:23.329855 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3492,"thread_start_us":122,"threads_started":1}
I20260929 18:37:23.330128 15378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:23.322641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.2:39877 (local address 127.13.100.1:45983)
0929 18:37:23.322784 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:23.322789 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:23.326239 (+ 3450us) server_negotiation.cc:407] Connection header received
0929 18:37:23.326381 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:23.326383 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:23.326436 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:23.326543 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:37:23.326823 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:23.327607 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:23.328847 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:23.329072 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:23.329162 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:23.329531 (+ 369us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:37:23.329602 (+ 71us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:37:23.329659 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:23.329875 (+ 216us) server_negotiation.cc:299] Negotiation successful
0929 18:37:23.329984 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 18:37:23.353379 14827 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48600:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:37:23.353545 14827 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48600:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:37:23.355640 14827 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8da0f949c56c4288b898ad0637fee260: Sending DeleteTablet for 3 replicas of tablet cbccb7dca15c4bdfa1618d176011a9aa
I20260929 18:37:23.356223 15106 tablet_service.cc:1514] Processing DeleteTablet for tablet cbccb7dca15c4bdfa1618d176011a9aa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:37:23 UTC) from {username='kudu', principal='kudu/127.13.100.62@KRBTEST.COM'} at 127.0.0.1:40824
I20260929 18:37:23.356364 15246 tablet_service.cc:1514] Processing DeleteTablet for tablet cbccb7dca15c4bdfa1618d176011a9aa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:37:23 UTC) from {username='kudu', principal='kudu/127.13.100.62@KRBTEST.COM'} at 127.0.0.1:45968
I20260929 18:37:23.356671 14966 tablet_service.cc:1514] Processing DeleteTablet for tablet cbccb7dca15c4bdfa1618d176011a9aa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:37:23 UTC) from {username='kudu', principal='kudu/127.13.100.62@KRBTEST.COM'} at 127.0.0.1:53896
I20260929 18:37:23.356959 15407 tablet_replica.cc:330] T cbccb7dca15c4bdfa1618d176011a9aa P 945f5bbc44574d57a45b266d384d2c59: stopping tablet replica
I20260929 18:37:23.357076 15407 raft_consensus.cc:2229] T cbccb7dca15c4bdfa1618d176011a9aa P 945f5bbc44574d57a45b266d384d2c59 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:37:23.357187 15407 raft_consensus.cc:2258] T cbccb7dca15c4bdfa1618d176011a9aa P 945f5bbc44574d57a45b266d384d2c59 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:37:23.357923 15407 ts_tablet_manager.cc:1905] T cbccb7dca15c4bdfa1618d176011a9aa P 945f5bbc44574d57a45b266d384d2c59: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:37:23.358137 15406 tablet_replica.cc:330] T cbccb7dca15c4bdfa1618d176011a9aa P 87b61322896e4e0c988d10faf801c9b5: stopping tablet replica
I20260929 18:37:23.358774 15406 raft_consensus.cc:2229] T cbccb7dca15c4bdfa1618d176011a9aa P 87b61322896e4e0c988d10faf801c9b5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:37:23.359273 15406 raft_consensus.cc:2258] T cbccb7dca15c4bdfa1618d176011a9aa P 87b61322896e4e0c988d10faf801c9b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:37:23.359941 15406 ts_tablet_manager.cc:1905] T cbccb7dca15c4bdfa1618d176011a9aa P 87b61322896e4e0c988d10faf801c9b5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:37:23.360737 15407 ts_tablet_manager.cc:1918] T cbccb7dca15c4bdfa1618d176011a9aa P 945f5bbc44574d57a45b266d384d2c59: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 18:37:23.360816 15407 log.cc:1196] T cbccb7dca15c4bdfa1618d176011a9aa P 945f5bbc44574d57a45b266d384d2c59: Deleting WAL directory at /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707036041078-13712-0/minicluster-data/ts-2/wal/wals/cbccb7dca15c4bdfa1618d176011a9aa
I20260929 18:37:23.361073 15407 ts_tablet_manager.cc:1939] T cbccb7dca15c4bdfa1618d176011a9aa P 945f5bbc44574d57a45b266d384d2c59: Deleting consensus metadata
I20260929 18:37:23.361719 14814 catalog_manager.cc:4841] TS 945f5bbc44574d57a45b266d384d2c59 (127.13.100.3:32827): tablet cbccb7dca15c4bdfa1618d176011a9aa (table test-table [id=c1e847bffa5a4175972ab0a3e0ff0243]) successfully deleted
I20260929 18:37:23.362048 15408 tablet_replica.cc:330] T cbccb7dca15c4bdfa1618d176011a9aa P fdfd0eb5ab40468594d7cb793150beb8: stopping tablet replica
I20260929 18:37:23.362181 15408 raft_consensus.cc:2229] T cbccb7dca15c4bdfa1618d176011a9aa P fdfd0eb5ab40468594d7cb793150beb8 [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:37:23.362474 15408 raft_consensus.cc:2258] T cbccb7dca15c4bdfa1618d176011a9aa P fdfd0eb5ab40468594d7cb793150beb8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:37:23.363070 15408 ts_tablet_manager.cc:1905] T cbccb7dca15c4bdfa1618d176011a9aa P fdfd0eb5ab40468594d7cb793150beb8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:37:23.363273 15406 ts_tablet_manager.cc:1918] T cbccb7dca15c4bdfa1618d176011a9aa P 87b61322896e4e0c988d10faf801c9b5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 18:37:23.363431 15406 log.cc:1196] T cbccb7dca15c4bdfa1618d176011a9aa P 87b61322896e4e0c988d10faf801c9b5: Deleting WAL directory at /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707036041078-13712-0/minicluster-data/ts-1/wal/wals/cbccb7dca15c4bdfa1618d176011a9aa
I20260929 18:37:23.363745 15406 ts_tablet_manager.cc:1939] T cbccb7dca15c4bdfa1618d176011a9aa P 87b61322896e4e0c988d10faf801c9b5: Deleting consensus metadata
I20260929 18:37:23.364863 14815 catalog_manager.cc:4841] TS 87b61322896e4e0c988d10faf801c9b5 (127.13.100.2:32859): tablet cbccb7dca15c4bdfa1618d176011a9aa (table test-table [id=c1e847bffa5a4175972ab0a3e0ff0243]) successfully deleted
I20260929 18:37:23.366014 15408 ts_tablet_manager.cc:1918] T cbccb7dca15c4bdfa1618d176011a9aa P fdfd0eb5ab40468594d7cb793150beb8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 18:37:23.366092 15408 log.cc:1196] T cbccb7dca15c4bdfa1618d176011a9aa P fdfd0eb5ab40468594d7cb793150beb8: Deleting WAL directory at /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707036041078-13712-0/minicluster-data/ts-0/wal/wals/cbccb7dca15c4bdfa1618d176011a9aa
I20260929 18:37:23.366374 15408 ts_tablet_manager.cc:1939] T cbccb7dca15c4bdfa1618d176011a9aa P fdfd0eb5ab40468594d7cb793150beb8: Deleting consensus metadata
I20260929 18:37:23.366988 14814 catalog_manager.cc:4841] TS fdfd0eb5ab40468594d7cb793150beb8 (127.13.100.1:45983): tablet cbccb7dca15c4bdfa1618d176011a9aa (table test-table [id=c1e847bffa5a4175972ab0a3e0ff0243]) successfully deleted
Sep 29 18:37:23 dist-test-slave-j1xs krb5kdc[14784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707042, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.13.100.1@KRBTEST.COM
I20260929 18:37:23.377223 15378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:23.358569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53914 (local address 127.13.100.1:45983)
0929 18:37:23.358673 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:23.358677 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:23.365274 (+ 6597us) server_negotiation.cc:407] Connection header received
0929 18:37:23.365456 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:23.365459 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:23.365506 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:23.365556 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:23.366700 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:23.367487 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:23.368763 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:23.369114 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:23.369227 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:23.373034 (+ 3807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:23.373043 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:23.373057 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:37:23.373083 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:23.375131 (+ 2048us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:23.375677 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:23.375683 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:23.375685 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:23.375794 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:23.376107 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:23.376110 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:23.376112 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:23.376394 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:23.376648 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:23.376787 (+ 139us) server_negotiation.cc:299] Negotiation successful
0929 18:37:23.376912 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
W20260929 18:37:23.377626 15026 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53914
I20260929 18:37:23.388868 14878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:23.379324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48626 (local address 127.13.100.62:34301)
0929 18:37:23.379443 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:23.379447 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:23.380311 (+ 864us) server_negotiation.cc:407] Connection header received
0929 18:37:23.380404 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:23.380408 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:23.380459 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:23.380545 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:23.381327 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:23.382134 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:23.383291 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:23.383642 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:23.383714 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:23.385363 (+ 1649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:23.385367 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:23.385370 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:37:23.385396 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:23.387069 (+ 1673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:23.387537 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:23.387543 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:23.387546 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:23.387671 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:23.387984 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:23.387988 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:23.387990 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:23.388204 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:23.388333 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:23.388575 (+ 242us) server_negotiation.cc:299] Negotiation successful
0929 18:37:23.388719 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
W20260929 18:37:23.389659 14827 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48626
I20260929 18:37:23.391191 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 14901
I20260929 18:37:23.400339 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 15041
I20260929 18:37:23.405759 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 15181
I20260929 18:37:23.412134 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 14797
2026-09-29T18:37:23Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2848 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707036041078-13712-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:37:23 dist-test-slave-j1xs krb5kdc[15415](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:37:23 dist-test-slave-j1xs krb5kdc[15415](info): set up 2 sockets
Sep 29 18:37:23 dist-test-slave-j1xs krb5kdc[15415](info): commencing operation
krb5kdc: starting...
W20260929 18:37:24.445711 13712 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707036041078-13712-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707036041078-13712-0/krb5kdc/test-user.keytab.
Sep 29 18:37:24 dist-test-slave-j1xs krb5kdc[15415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707044, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:37:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:37:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.100.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.100.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.100.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.62.keytab.
Entry for principal kudu/127.13.100.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.62.keytab.
I20260929 18:37:24.565750 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707036041078-13712-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707036041078-13712-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707036041078-13712-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.100.62:43705
--webserver_interface=127.13.100.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.62.keytab
--principal=kudu/127.13.100.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.100.20:46007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.100.62:43705
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707036041078-13712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707036041078-13712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707036041078-13712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:37:24.674793 15428 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:37:24.675035 15428 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:37:24.675074 15428 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:37:24.678979 15428 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:37:24.679028 15428 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 18:37:24.679044 15428 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:37:24.679062 15428 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:37:24.679077 15428 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:37:24.679092 15428 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:37:24.684216 15428 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.100.20:46007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707036041078-13712-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707036041078-13712-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.100.62:43705
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.62.keytab
--principal=kudu/127.13.100.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.100.62:43705
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.100.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15428
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707036041078-13712-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:37:24.685148 15428 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:37:24.686094 15428 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:37:24.694460 15434 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:37:24.694540 15436 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:37:24.694593 15428 server_base.cc:1034] running on GCE node
W20260929 18:37:24.694546 15433 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:37:24.695259 15428 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:37:24.696388 15428 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:37:24.697571 15428 hybrid_clock.cc:648] HybridClock initialized: now 1790707044697538 us; error 40 us; skew 500 ppm
Sep 29 18:37:24 dist-test-slave-j1xs krb5kdc[15415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707044, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:37:24.700455 15428 init.cc:377] Logged in from keytab as kudu/127.13.100.62@KRBTEST.COM (short username kudu)
I20260929 18:37:24.701248 15428 webserver.cc:460] Webserver started at http://127.13.100.62:43411/ using document root <none> and password file <none>
I20260929 18:37:24.701737 15428 fs_manager.cc:359] Metadata directory not provided
I20260929 18:37:24.701794 15428 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:37:24.702025 15428 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:37:24.704003 15428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707036041078-13712-0/minicluster-data/master-0/data/instance:
uuid: "bb338bdc225b47fca102578584452513"
format_stamp: "Formatted at 2026-09-29 18:37:24 on dist-test-slave-j1xs"
server_key: "a3734c1a82f747f0b05adb3fea58d4ca"
server_key_iv: "e7ce2b3dbccaca5d47f55be080bf230a"
server_key_version: "encryptionkey@0"
I20260929 18:37:24.704619 15428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707036041078-13712-0/minicluster-data/master-0/wal/instance:
uuid: "bb338bdc225b47fca102578584452513"
format_stamp: "Formatted at 2026-09-29 18:37:24 on dist-test-slave-j1xs"
server_key: "a3734c1a82f747f0b05adb3fea58d4ca"
server_key_iv: "e7ce2b3dbccaca5d47f55be080bf230a"
server_key_version: "encryptionkey@0"
I20260929 18:37:24.708727 15428 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 18:37:24.711465 15443 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:37:24.712808 15428 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 18:37:24.712913 15428 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707036041078-13712-0/minicluster-data/master-0/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707036041078-13712-0/minicluster-data/master-0/wal
uuid: "bb338bdc225b47fca102578584452513"
format_stamp: "Formatted at 2026-09-29 18:37:24 on dist-test-slave-j1xs"
server_key: "a3734c1a82f747f0b05adb3fea58d4ca"
server_key_iv: "e7ce2b3dbccaca5d47f55be080bf230a"
server_key_version: "encryptionkey@0"
I20260929 18:37:24.713006 15428 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707036041078-13712-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707036041078-13712-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707036041078-13712-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:37:24.733456 15428 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:37:24.736629 15428 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:37:24.736840 15428 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:37:24.744776 15428 rpc_server.cc:307] RPC server started. Bound to: 127.13.100.62:43705
I20260929 18:37:24.744796 15505 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.100.62:43705 every 8 connection(s)
I20260929 18:37:24.745700 15428 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
I20260929 18:37:24.748390 15506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:37:24.752067 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 15428
I20260929 18:37:24.752153 13712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707036041078-13712-0/minicluster-data/master-0/wal/instance
I20260929 18:37:24.752416 13712 external_mini_cluster.cc:1428] Setting key 89596630a8dd6dda9a70f115c072fee0
I20260929 18:37:24.754439 15506 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb338bdc225b47fca102578584452513: Bootstrap starting.
I20260929 18:37:24.756526 15506 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bb338bdc225b47fca102578584452513: Neither blocks nor log segments found. Creating new log.
I20260929 18:37:24.757229 15506 log.cc:824] T 00000000000000000000000000000000 P bb338bdc225b47fca102578584452513: Log is configured to *not* fsync() on all Append() calls
I20260929 18:37:24.759047 15506 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb338bdc225b47fca102578584452513: No bootstrap required, opened a new log
Sep 29 18:37:24 dist-test-slave-j1xs krb5kdc[15415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707044, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.100.62@KRBTEST.COM
I20260929 18:37:24.762418 15506 raft_consensus.cc:348] T 00000000000000000000000000000000 P bb338bdc225b47fca102578584452513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb338bdc225b47fca102578584452513" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 43705 } }
I20260929 18:37:24.762653 15506 raft_consensus.cc:374] T 00000000000000000000000000000000 P bb338bdc225b47fca102578584452513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:37:24.762734 15506 raft_consensus.cc:729] T 00000000000000000000000000000000 P bb338bdc225b47fca102578584452513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb338bdc225b47fca102578584452513, State: Initialized, Role: FOLLOWER
I20260929 18:37:24.763201 15506 consensus_queue.cc:260] T 00000000000000000000000000000000 P bb338bdc225b47fca102578584452513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb338bdc225b47fca102578584452513" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 43705 } }
I20260929 18:37:24.763356 15506 raft_consensus.cc:388] T 00000000000000000000000000000000 P bb338bdc225b47fca102578584452513 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:37:24.763432 15506 raft_consensus.cc:482] T 00000000000000000000000000000000 P bb338bdc225b47fca102578584452513 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:37:24.763537 15506 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bb338bdc225b47fca102578584452513 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:37:24.764575 15506 raft_consensus.cc:504] T 00000000000000000000000000000000 P bb338bdc225b47fca102578584452513 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb338bdc225b47fca102578584452513" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 43705 } }
I20260929 18:37:24.764921 15506 leader_election.cc:302] T 00000000000000000000000000000000 P bb338bdc225b47fca102578584452513 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bb338bdc225b47fca102578584452513; no voters:
I20260929 18:37:24.765262 15506 leader_election.cc:288] T 00000000000000000000000000000000 P bb338bdc225b47fca102578584452513 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:37:24.765442 15511 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bb338bdc225b47fca102578584452513 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:37:24.765653 15511 raft_consensus.cc:686] T 00000000000000000000000000000000 P bb338bdc225b47fca102578584452513 [term 1 LEADER]: Becoming Leader. State: Replica: bb338bdc225b47fca102578584452513, State: Running, Role: LEADER
I20260929 18:37:24.765998 15511 consensus_queue.cc:237] T 00000000000000000000000000000000 P bb338bdc225b47fca102578584452513 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb338bdc225b47fca102578584452513" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 43705 } }
I20260929 18:37:24.766491 15509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:24.753798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39210 (local address 127.13.100.62:43705)
0929 18:37:24.754224 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:24.754239 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:24.754273 (+ 34us) server_negotiation.cc:407] Connection header received
0929 18:37:24.754941 (+ 668us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:24.754958 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:24.755271 (+ 313us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:24.755597 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:24.756364 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:24.757485 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:24.758376 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:24.758836 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:24.759013 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:24.761403 (+ 2390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:24.761409 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:24.761421 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:37:24.761446 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:24.763301 (+ 1855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:24.763860 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:24.763864 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:24.763868 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:24.763975 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:24.764256 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:24.764258 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:24.764259 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:24.764617 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:24.764813 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:24.765171 (+ 358us) server_negotiation.cc:299] Negotiation successful
0929 18:37:24.765490 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":128,"threads_started":1}
I20260929 18:37:24.766527 15506 sys_catalog.cc:564] T 00000000000000000000000000000000 P bb338bdc225b47fca102578584452513 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:37:24.767191 15511 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb338bdc225b47fca102578584452513 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bb338bdc225b47fca102578584452513. Latest consensus state: current_term: 1 leader_uuid: "bb338bdc225b47fca102578584452513" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb338bdc225b47fca102578584452513" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 43705 } } }
I20260929 18:37:24.767306 15511 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb338bdc225b47fca102578584452513 [sys.catalog]: This master's current role is: LEADER
I20260929 18:37:24.767759 15512 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb338bdc225b47fca102578584452513 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bb338bdc225b47fca102578584452513" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb338bdc225b47fca102578584452513" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 43705 } } }
I20260929 18:37:24.767867 15512 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb338bdc225b47fca102578584452513 [sys.catalog]: This master's current role is: LEADER
I20260929 18:37:24.768590 15517 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:37:24.771646 15517 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:37:24.777055 15517 catalog_manager.cc:1347] Generated new cluster ID: a464580f3c2f43f696ae9ca8e7176fba
I20260929 18:37:24.777118 15517 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:37:24.782624 15517 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:37:24.783771 15517 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:37:24.788698 15517 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bb338bdc225b47fca102578584452513: Generated new TSK 0
I20260929 18:37:24.789463 15517 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.100.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.100.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.100.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.1.keytab.
Entry for principal kudu/127.13.100.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.1.keytab.
I20260929 18:37:24.826857 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707036041078-13712-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707036041078-13712-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707036041078-13712-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707036041078-13712-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.100.1:0
--local_ip_for_outbound_sockets=127.13.100.1
--webserver_interface=127.13.100.1
--webserver_port=0
--tserver_master_addrs=127.13.100.62:43705
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.1.keytab
--principal=kudu/127.13.100.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.100.20:46007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707036041078-13712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707036041078-13712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707036041078-13712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:37:24.936215 15532 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:37:24.936498 15532 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:37:24.936578 15532 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:37:24.940502 15532 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:37:24.940587 15532 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:37:24.940659 15532 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 18:37:24.940704 15532 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 18:37:24.940770 15532 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.100.1
I20260929 18:37:24.945878 15532 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.100.20:46007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707036041078-13712-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707036041078-13712-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.1.keytab
--principal=kudu/127.13.100.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.100.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707036041078-13712-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.100.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.13.100.62:43705
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15532
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.100.1
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707036041078-13712-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:37:24.946930 15532 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:37:24.947963 15532 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:37:24.957325 15537 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:37:24.957340 15540 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:37:24.957410 15538 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:37:24.957573 15532 server_base.cc:1034] running on GCE node
I20260929 18:37:24.958068 15532 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:37:24.958730 15532 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:37:24.959914 15532 hybrid_clock.cc:648] HybridClock initialized: now 1790707044959898 us; error 30 us; skew 500 ppm
Sep 29 18:37:24 dist-test-slave-j1xs krb5kdc[15415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707044, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:37:24.962692 15532 init.cc:377] Logged in from keytab as kudu/127.13.100.1@KRBTEST.COM (short username kudu)
I20260929 18:37:24.963575 15532 webserver.cc:460] Webserver started at http://127.13.100.1:43635/ using document root <none> and password file <none>
I20260929 18:37:24.964176 15532 fs_manager.cc:359] Metadata directory not provided
I20260929 18:37:24.964264 15532 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:37:24.964490 15532 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:37:24.967013 15532 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707036041078-13712-0/minicluster-data/ts-0/data/instance:
uuid: "f5f5cca3d04940e781e1bb7af2217ae1"
format_stamp: "Formatted at 2026-09-29 18:37:24 on dist-test-slave-j1xs"
server_key: "70fd1e9958961ec5d2f716e1065f61d6"
server_key_iv: "e36a463b80fe463d67ba145a2003f694"
server_key_version: "encryptionkey@0"
I20260929 18:37:24.967641 15532 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707036041078-13712-0/minicluster-data/ts-0/wal/instance:
uuid: "f5f5cca3d04940e781e1bb7af2217ae1"
format_stamp: "Formatted at 2026-09-29 18:37:24 on dist-test-slave-j1xs"
server_key: "70fd1e9958961ec5d2f716e1065f61d6"
server_key_iv: "e36a463b80fe463d67ba145a2003f694"
server_key_version: "encryptionkey@0"
I20260929 18:37:24.971570 15532 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260929 18:37:24.974285 15547 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:37:24.975492 15532 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 18:37:24.975623 15532 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707036041078-13712-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707036041078-13712-0/minicluster-data/ts-0/wal
uuid: "f5f5cca3d04940e781e1bb7af2217ae1"
format_stamp: "Formatted at 2026-09-29 18:37:24 on dist-test-slave-j1xs"
server_key: "70fd1e9958961ec5d2f716e1065f61d6"
server_key_iv: "e36a463b80fe463d67ba145a2003f694"
server_key_version: "encryptionkey@0"
I20260929 18:37:24.975737 15532 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707036041078-13712-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707036041078-13712-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707036041078-13712-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:37:24.995527 15532 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:37:24.998960 15532 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:37:24.999168 15532 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:37:25.000581 15532 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:37:25.000656 15532 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:37:25.000728 15532 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:37:25.000782 15532 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 18:37:25 dist-test-slave-j1xs krb5kdc[15415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707044, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.1@KRBTEST.COM for kudu/127.13.100.62@KRBTEST.COM
I20260929 18:37:25.011615 15532 rpc_server.cc:307] RPC server started. Bound to: 127.13.100.1:46433
I20260929 18:37:25.011677 15662 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.100.1:46433 every 8 connection(s)
I20260929 18:37:25.012655 15532 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707036041078-13712-0/minicluster-data/ts-0/data/info.pb
I20260929 18:37:25.013329 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 15532
I20260929 18:37:25.013453 13712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707036041078-13712-0/minicluster-data/ts-0/wal/instance
I20260929 18:37:25.013790 13712 external_mini_cluster.cc:1428] Setting key 5ad734b372bc34eff8dd3ccb2c754bfc
I20260929 18:37:25.015974 15509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:25.001707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.1:46359 (local address 127.13.100.62:43705)
0929 18:37:25.001849 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:25.001853 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:25.002979 (+ 1126us) server_negotiation.cc:407] Connection header received
0929 18:37:25.004314 (+ 1335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:25.004320 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:25.004395 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:25.004482 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:25.006115 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:25.006901 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:25.008115 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:25.008367 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:25.008491 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:25.011791 (+ 3300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:25.011798 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:25.011801 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:37:25.011831 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:25.013637 (+ 1806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:25.014286 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:25.014290 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:25.014293 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:25.014399 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:25.014776 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:25.014779 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:25.014780 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:25.014978 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:25.015076 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:25.015677 (+ 601us) server_negotiation.cc:299] Negotiation successful
0929 18:37:25.015786 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 18:37:25.017060 15556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:25.002274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:43705 (local address 127.13.100.1:46359)
0929 18:37:25.002780 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:25.002827 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:25.004056 (+ 1229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:25.004716 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:25.004728 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:25.005238 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:37:25.005951 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:25.005970 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:25.007020 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:25.007024 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:25.007986 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:25.007996 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:25.008538 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:25.008543 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:25.008740 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:25.009398 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:37:25.009420 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:37:25.011549 (+ 2129us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:37:25.013763 (+ 2214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:25.013769 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:25.013782 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:25.014158 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:25.014493 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:25.014496 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:25.014498 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:25.014678 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:25.015097 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:37:25.015104 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:37:25.015418 (+ 314us) client_negotiation.cc:769] Sending connection context
0929 18:37:25.015614 (+ 196us) client_negotiation.cc:240] Negotiation successful
0929 18:37:25.015981 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":125,"threads_started":1}
I20260929 18:37:25.018497 15663 heartbeater.cc:347] Connected to a master server at 127.13.100.62:43705
I20260929 18:37:25.018642 15663 heartbeater.cc:464] Registering TS with master...
I20260929 18:37:25.019187 15663 heartbeater.cc:511] Master 127.13.100.62:43705 requested a full tablet report, sending...
I20260929 18:37:25.020845 15460 ts_manager.cc:194] Registered new tserver with Master: f5f5cca3d04940e781e1bb7af2217ae1 (127.13.100.1:46433)
I20260929 18:37:25.022584 15460 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.100.1@KRBTEST.COM'} at 127.13.100.1:46359
WARNING: no policy specified for kudu/127.13.100.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.100.2@KRBTEST.COM" created.
I20260929 18:37:25.027945 15670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:25.020533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:43705 (local address 127.13.100.1:56219)
0929 18:37:25.020821 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:25.020832 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:25.020938 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:25.021157 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:25.021160 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:25.021363 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:37:25.021456 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:25.021461 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:25.022515 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:25.022519 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:25.023083 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:25.023092 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:25.023749 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:25.023751 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:25.023820 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:25.024359 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:37:25.024370 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:37:25.024749 (+ 379us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:37:25.026527 (+ 1778us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:25.026533 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:25.026535 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:25.026734 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:25.027009 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:25.027013 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:25.027015 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:25.027103 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:25.027501 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:37:25.027503 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:37:25.027579 (+ 76us) client_negotiation.cc:769] Sending connection context
0929 18:37:25.027668 (+ 89us) client_negotiation.cc:240] Negotiation successful
0929 18:37:25.027773 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":86,"threads_started":1}
I20260929 18:37:25.028155 15509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:25.020579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.1:56219 (local address 127.13.100.62:43705)
0929 18:37:25.020690 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:25.020694 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:25.020901 (+ 207us) server_negotiation.cc:407] Connection header received
0929 18:37:25.021012 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:25.021015 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:25.021054 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:25.021145 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:25.021567 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:25.022395 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:25.023247 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:25.023562 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:25.023623 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:25.024903 (+ 1280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:25.024931 (+ 28us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:25.024934 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:37:25.024954 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:25.026421 (+ 1467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:25.026830 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:25.026835 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:25.026837 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:25.026917 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:25.027178 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:25.027183 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:25.027185 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:25.027396 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:25.027504 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:25.027762 (+ 258us) server_negotiation.cc:299] Negotiation successful
0929 18:37:25.027899 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":189,"server-negotiator.queue_time_us":45}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.100.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.2.keytab.
Entry for principal kudu/127.13.100.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.2.keytab.
I20260929 18:37:25.041134 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707036041078-13712-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707036041078-13712-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707036041078-13712-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707036041078-13712-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.100.2:0
--local_ip_for_outbound_sockets=127.13.100.2
--webserver_interface=127.13.100.2
--webserver_port=0
--tserver_master_addrs=127.13.100.62:43705
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.2.keytab
--principal=kudu/127.13.100.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.100.20:46007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707036041078-13712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707036041078-13712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707036041078-13712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:37:25.155927 15672 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:37:25.156150 15672 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:37:25.156195 15672 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:37:25.160192 15672 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:37:25.160246 15672 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:37:25.160293 15672 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 18:37:25.160307 15672 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 18:37:25.160339 15672 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.100.2
I20260929 18:37:25.165242 15672 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.100.20:46007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707036041078-13712-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707036041078-13712-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.2.keytab
--principal=kudu/127.13.100.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.100.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707036041078-13712-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.100.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.13.100.62:43705
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15672
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.100.2
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707036041078-13712-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:37:25.166227 15672 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:37:25.167275 15672 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:37:25.176523 15677 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:37:25.176671 15672 server_base.cc:1034] running on GCE node
W20260929 18:37:25.176501 15678 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:37:25.176496 15680 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:37:25.177145 15672 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:37:25.177728 15672 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:37:25.178885 15672 hybrid_clock.cc:648] HybridClock initialized: now 1790707045178873 us; error 34 us; skew 500 ppm
Sep 29 18:37:25 dist-test-slave-j1xs krb5kdc[15415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707045, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:37:25.181433 15672 init.cc:377] Logged in from keytab as kudu/127.13.100.2@KRBTEST.COM (short username kudu)
I20260929 18:37:25.182224 15672 webserver.cc:460] Webserver started at http://127.13.100.2:43781/ using document root <none> and password file <none>
I20260929 18:37:25.182695 15672 fs_manager.cc:359] Metadata directory not provided
I20260929 18:37:25.182749 15672 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:37:25.182907 15672 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:37:25.184737 15672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707036041078-13712-0/minicluster-data/ts-1/data/instance:
uuid: "601f7285b75f48539695ba51f9db205b"
format_stamp: "Formatted at 2026-09-29 18:37:25 on dist-test-slave-j1xs"
server_key: "330b82c8c34156373c459e76c14f68c0"
server_key_iv: "6ba758fe10cf1420310ae0219ac05888"
server_key_version: "encryptionkey@0"
I20260929 18:37:25.185217 15672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707036041078-13712-0/minicluster-data/ts-1/wal/instance:
uuid: "601f7285b75f48539695ba51f9db205b"
format_stamp: "Formatted at 2026-09-29 18:37:25 on dist-test-slave-j1xs"
server_key: "330b82c8c34156373c459e76c14f68c0"
server_key_iv: "6ba758fe10cf1420310ae0219ac05888"
server_key_version: "encryptionkey@0"
I20260929 18:37:25.188671 15672 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260929 18:37:25.190972 15687 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:37:25.192150 15672 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:37:25.192279 15672 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707036041078-13712-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707036041078-13712-0/minicluster-data/ts-1/wal
uuid: "601f7285b75f48539695ba51f9db205b"
format_stamp: "Formatted at 2026-09-29 18:37:25 on dist-test-slave-j1xs"
server_key: "330b82c8c34156373c459e76c14f68c0"
server_key_iv: "6ba758fe10cf1420310ae0219ac05888"
server_key_version: "encryptionkey@0"
I20260929 18:37:25.192389 15672 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707036041078-13712-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707036041078-13712-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707036041078-13712-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:37:25.204658 15672 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:37:25.207954 15672 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:37:25.208149 15672 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:37:25.209529 15672 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:37:25.209575 15672 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:37:25.209617 15672 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:37:25.209673 15672 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 18:37:25 dist-test-slave-j1xs krb5kdc[15415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707045, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.2@KRBTEST.COM for kudu/127.13.100.62@KRBTEST.COM
I20260929 18:37:25.220623 15672 rpc_server.cc:307] RPC server started. Bound to: 127.13.100.2:35501
I20260929 18:37:25.220683 15802 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.100.2:35501 every 8 connection(s)
I20260929 18:37:25.221817 15672 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707036041078-13712-0/minicluster-data/ts-1/data/info.pb
I20260929 18:37:25.223994 15509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:25.210389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.2:60735 (local address 127.13.100.62:43705)
0929 18:37:25.210518 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:25.210525 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:25.211524 (+ 999us) server_negotiation.cc:407] Connection header received
0929 18:37:25.212557 (+ 1033us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:25.212562 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:25.212649 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:25.212717 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:25.214491 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:25.215065 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:25.216771 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:25.217158 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:25.217274 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:25.220192 (+ 2918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:25.220196 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:25.220199 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:37:25.220228 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:25.222046 (+ 1818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:25.222635 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:25.222642 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:25.222647 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:25.222746 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:25.223077 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:25.223079 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:25.223081 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:25.223258 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:25.223335 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:25.223766 (+ 431us) server_negotiation.cc:299] Negotiation successful
0929 18:37:25.223876 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 18:37:25.224750 15696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:25.210938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:43705 (local address 127.13.100.2:60735)
0929 18:37:25.211384 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:25.211424 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:25.212305 (+ 881us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:25.213188 (+ 883us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:25.213196 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:25.213625 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:37:25.214303 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:25.214324 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:25.215194 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:25.215197 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:25.216654 (+ 1457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:25.216666 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:25.217318 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:25.217322 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:25.217516 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:25.218153 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:37:25.218169 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:37:25.220018 (+ 1849us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:37:25.222152 (+ 2134us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:25.222156 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:25.222183 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:25.222510 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:25.222858 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:25.222861 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:25.222862 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:25.222987 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:25.223346 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:37:25.223350 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:37:25.223554 (+ 204us) client_negotiation.cc:769] Sending connection context
0929 18:37:25.223722 (+ 168us) client_negotiation.cc:240] Negotiation successful
0929 18:37:25.223953 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":124,"threads_started":1}
I20260929 18:37:25.225971 15803 heartbeater.cc:347] Connected to a master server at 127.13.100.62:43705
I20260929 18:37:25.226135 15803 heartbeater.cc:464] Registering TS with master...
I20260929 18:37:25.226634 15803 heartbeater.cc:511] Master 127.13.100.62:43705 requested a full tablet report, sending...
I20260929 18:37:25.227432 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 15672
I20260929 18:37:25.227551 13712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707036041078-13712-0/minicluster-data/ts-1/wal/instance
I20260929 18:37:25.227701 15459 ts_manager.cc:194] Registered new tserver with Master: 601f7285b75f48539695ba51f9db205b (127.13.100.2:35501)
I20260929 18:37:25.227901 13712 external_mini_cluster.cc:1428] Setting key 1921a8e2e96b7c1d166fb45ceb6542ea
I20260929 18:37:25.228495 15459 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.100.2@KRBTEST.COM'} at 127.13.100.2:60735
I20260929 18:37:25.235810 15509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:25.228074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.2:55033 (local address 127.13.100.62:43705)
0929 18:37:25.228172 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:25.228176 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:25.228404 (+ 228us) server_negotiation.cc:407] Connection header received
0929 18:37:25.228494 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:25.228496 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:25.228535 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:25.228640 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:25.229176 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:25.229995 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:25.231245 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:25.231452 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:25.231574 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:25.232646 (+ 1072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:25.232650 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:25.232653 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:37:25.232683 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:25.234195 (+ 1512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:25.234622 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:25.234627 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:25.234629 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:25.234707 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:25.235003 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:25.235006 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:25.235008 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:25.235202 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:25.235310 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:25.235568 (+ 258us) server_negotiation.cc:299] Negotiation successful
0929 18:37:25.235672 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":39}
I20260929 18:37:25.235837 15809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:25.228015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:43705 (local address 127.13.100.2:55033)
0929 18:37:25.228310 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:25.228323 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:25.228414 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:25.228695 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:25.228698 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:25.228922 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:37:25.229057 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:25.229065 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:25.230139 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:25.230142 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:25.231118 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:25.231132 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:25.231578 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:25.231582 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:25.231674 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:25.232185 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:37:25.232196 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:37:25.232498 (+ 302us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:37:25.234305 (+ 1807us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:25.234311 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:25.234313 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:25.234513 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:25.234807 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:25.234809 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:25.234811 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:25.234903 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:25.235309 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:37:25.235312 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:37:25.235404 (+ 92us) client_negotiation.cc:769] Sending connection context
0929 18:37:25.235506 (+ 102us) client_negotiation.cc:240] Negotiation successful
0929 18:37:25.235642 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":88,"threads_started":1}
WARNING: no policy specified for kudu/127.13.100.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.100.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.100.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.3.keytab.
Entry for principal kudu/127.13.100.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.3.keytab.
I20260929 18:37:25.254611 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707036041078-13712-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707036041078-13712-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707036041078-13712-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707036041078-13712-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.100.3:0
--local_ip_for_outbound_sockets=127.13.100.3
--webserver_interface=127.13.100.3
--webserver_port=0
--tserver_master_addrs=127.13.100.62:43705
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.3.keytab
--principal=kudu/127.13.100.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.100.20:46007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707036041078-13712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707036041078-13712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707036041078-13712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:37:25.366750 15812 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:37:25.367070 15812 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:37:25.367156 15812 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:37:25.371578 15812 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:37:25.371670 15812 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:37:25.371740 15812 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 18:37:25.371780 15812 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 18:37:25.371857 15812 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.100.3
I20260929 18:37:25.377156 15812 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.100.20:46007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707036041078-13712-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707036041078-13712-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.3.keytab
--principal=kudu/127.13.100.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.100.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707036041078-13712-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.100.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.13.100.62:43705
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15812
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.100.3
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707036041078-13712-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:37:25.378252 15812 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:37:25.379293 15812 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:37:25.388890 15818 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:37:25.388859 15817 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:37:25.389045 15820 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:37:25.389158 15812 server_base.cc:1034] running on GCE node
I20260929 18:37:25.389532 15812 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:37:25.390213 15812 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:37:25.391373 15812 hybrid_clock.cc:648] HybridClock initialized: now 1790707045391354 us; error 32 us; skew 500 ppm
Sep 29 18:37:25 dist-test-slave-j1xs krb5kdc[15415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707045, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:37:25.395015 15812 init.cc:377] Logged in from keytab as kudu/127.13.100.3@KRBTEST.COM (short username kudu)
I20260929 18:37:25.395900 15812 webserver.cc:460] Webserver started at http://127.13.100.3:42831/ using document root <none> and password file <none>
I20260929 18:37:25.396528 15812 fs_manager.cc:359] Metadata directory not provided
I20260929 18:37:25.396620 15812 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:37:25.396833 15812 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:37:25.398921 15812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707036041078-13712-0/minicluster-data/ts-2/data/instance:
uuid: "16fcd7a8ee8a47d2926ab4f3db9b48f6"
format_stamp: "Formatted at 2026-09-29 18:37:25 on dist-test-slave-j1xs"
server_key: "7f8044d724207b354d5d40bb9379d15c"
server_key_iv: "16f937cb646c95b47692c409538e00f9"
server_key_version: "encryptionkey@0"
I20260929 18:37:25.399484 15812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707036041078-13712-0/minicluster-data/ts-2/wal/instance:
uuid: "16fcd7a8ee8a47d2926ab4f3db9b48f6"
format_stamp: "Formatted at 2026-09-29 18:37:25 on dist-test-slave-j1xs"
server_key: "7f8044d724207b354d5d40bb9379d15c"
server_key_iv: "16f937cb646c95b47692c409538e00f9"
server_key_version: "encryptionkey@0"
I20260929 18:37:25.403357 15812 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 18:37:25.406304 15827 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:37:25.407562 15812 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:37:25.407706 15812 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707036041078-13712-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707036041078-13712-0/minicluster-data/ts-2/wal
uuid: "16fcd7a8ee8a47d2926ab4f3db9b48f6"
format_stamp: "Formatted at 2026-09-29 18:37:25 on dist-test-slave-j1xs"
server_key: "7f8044d724207b354d5d40bb9379d15c"
server_key_iv: "16f937cb646c95b47692c409538e00f9"
server_key_version: "encryptionkey@0"
I20260929 18:37:25.407827 15812 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707036041078-13712-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707036041078-13712-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707036041078-13712-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:37:25.420626 15812 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:37:25.423336 15812 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:37:25.423524 15812 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:37:25.424829 15812 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:37:25.424901 15812 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:37:25.424968 15812 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:37:25.425019 15812 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 18:37:25 dist-test-slave-j1xs krb5kdc[15415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707045, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.3@KRBTEST.COM for kudu/127.13.100.62@KRBTEST.COM
I20260929 18:37:25.435611 15812 rpc_server.cc:307] RPC server started. Bound to: 127.13.100.3:35143
I20260929 18:37:25.435753 15942 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.100.3:35143 every 8 connection(s)
I20260929 18:37:25.436687 15812 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707036041078-13712-0/minicluster-data/ts-2/data/info.pb
I20260929 18:37:25.439122 15509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:25.425730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.3:51283 (local address 127.13.100.62:43705)
0929 18:37:25.425854 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:25.425858 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:25.426952 (+ 1094us) server_negotiation.cc:407] Connection header received
0929 18:37:25.428163 (+ 1211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:25.428166 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:25.428221 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:25.428326 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:25.429835 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:25.430682 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:25.432515 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:25.432781 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:25.432869 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:25.435700 (+ 2831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:25.435704 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:25.435707 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:37:25.435733 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:25.437284 (+ 1551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:25.437841 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:25.437844 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:25.437845 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:25.437967 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:25.438237 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:25.438240 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:25.438241 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:25.438419 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:25.438496 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:25.438919 (+ 423us) server_negotiation.cc:299] Negotiation successful
0929 18:37:25.439010 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 18:37:25.439898 15836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:25.426282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:43705 (local address 127.13.100.3:51283)
0929 18:37:25.426818 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:25.426857 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:25.427950 (+ 1093us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:25.428493 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:25.428501 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:25.428954 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:37:25.429629 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:25.429647 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:25.430835 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:25.430842 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:25.432372 (+ 1530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:25.432386 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:25.432963 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:25.432968 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:25.433180 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:25.433867 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:37:25.433911 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:37:25.435526 (+ 1615us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:37:25.437420 (+ 1894us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:25.437427 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:25.437439 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:25.437749 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:25.438036 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:25.438038 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:25.438040 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:25.438153 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:25.438507 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:37:25.438511 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:37:25.438722 (+ 211us) client_negotiation.cc:769] Sending connection context
0929 18:37:25.438874 (+ 152us) client_negotiation.cc:240] Negotiation successful
0929 18:37:25.439088 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":121,"threads_started":1}
I20260929 18:37:25.440598 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 15812
I20260929 18:37:25.440711 13712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707036041078-13712-0/minicluster-data/ts-2/wal/instance
I20260929 18:37:25.440999 13712 external_mini_cluster.cc:1428] Setting key 55aa6efd0e0a511f67776a91b953fb76
I20260929 18:37:25.441258 15943 heartbeater.cc:347] Connected to a master server at 127.13.100.62:43705
I20260929 18:37:25.441399 15943 heartbeater.cc:464] Registering TS with master...
I20260929 18:37:25.442020 15943 heartbeater.cc:511] Master 127.13.100.62:43705 requested a full tablet report, sending...
I20260929 18:37:25.443030 15459 ts_manager.cc:194] Registered new tserver with Master: 16fcd7a8ee8a47d2926ab4f3db9b48f6 (127.13.100.3:35143)
I20260929 18:37:25.443748 15459 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.100.3@KRBTEST.COM'} at 127.13.100.3:51283
I20260929 18:37:25.449723 15509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:25.443447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.3:58831 (local address 127.13.100.62:43705)
0929 18:37:25.443576 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:25.443580 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:25.443709 (+ 129us) server_negotiation.cc:407] Connection header received
0929 18:37:25.443841 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:25.443843 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:25.443883 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:25.443970 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:25.444360 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:25.444843 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:25.445621 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:25.445850 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:25.446001 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:25.447007 (+ 1006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:25.447011 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:25.447013 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:37:25.447036 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:25.448195 (+ 1159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:25.448596 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:25.448601 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:25.448602 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:25.448681 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:25.448967 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:25.448970 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:25.448971 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:25.449134 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:25.449236 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:25.449490 (+ 254us) server_negotiation.cc:299] Negotiation successful
0929 18:37:25.449591 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 18:37:25.449735 15949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:25.443373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:43705 (local address 127.13.100.3:58831)
0929 18:37:25.443618 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:25.443631 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:25.443745 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:25.443982 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:25.443984 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:25.444167 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:37:25.444253 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:25.444258 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:25.444960 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:25.444963 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:25.445508 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:25.445515 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:25.446003 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:25.446007 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:25.446096 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:25.446573 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:37:25.446589 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:37:25.446870 (+ 281us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:37:25.448310 (+ 1440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:25.448313 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:25.448315 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:25.448499 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:25.448780 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:25.448783 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:25.448784 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:25.448871 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:25.449226 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:37:25.449229 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:37:25.449313 (+ 84us) client_negotiation.cc:769] Sending connection context
0929 18:37:25.449428 (+ 115us) client_negotiation.cc:240] Negotiation successful
0929 18:37:25.449557 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":80,"threads_started":1}
I20260929 18:37:25.454645 13712 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 18:37:25 dist-test-slave-j1xs krb5kdc[15415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707045, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 18:37:25 dist-test-slave-j1xs krb5kdc[15415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707045, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.100.62@KRBTEST.COM
I20260929 18:37:25.479156 15509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:25.471193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39216 (local address 127.13.100.62:43705)
0929 18:37:25.471319 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:25.471323 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:25.471335 (+ 12us) server_negotiation.cc:407] Connection header received
0929 18:37:25.471374 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:25.471377 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:25.471416 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:25.471492 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:25.472151 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:25.472663 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:25.473555 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:25.473771 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:25.473888 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:25.476219 (+ 2331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:25.476223 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:25.476226 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:37:25.476257 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:25.477482 (+ 1225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:25.477926 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:25.477930 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:25.477932 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:25.478011 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:25.478311 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:25.478314 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:25.478316 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:25.478491 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:25.478651 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:25.478875 (+ 224us) server_negotiation.cc:299] Negotiation successful
0929 18:37:25.478986 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 18:37:25.481827 15459 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39216:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:37:25.484764 15459 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:37:25.498612 15960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:25.492753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.2:35501 (local address 127.0.0.1:41276)
0929 18:37:25.493151 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:25.493206 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:25.493329 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:25.493970 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:25.493980 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:25.494007 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:37:25.494158 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:25.494165 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:25.495734 (+ 1569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:25.495737 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:25.497092 (+ 1355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:25.497104 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:25.498064 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:25.498068 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:25.498171 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:25.498211 (+ 40us) client_negotiation.cc:769] Sending connection context
0929 18:37:25.498318 (+ 107us) client_negotiation.cc:240] Negotiation successful
0929 18:37:25.498456 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":111,"threads_started":1}
I20260929 18:37:25.498646 15961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:25.492885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41276 (local address 127.13.100.2:35501)
0929 18:37:25.493325 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:25.493337 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:25.493362 (+ 25us) server_negotiation.cc:407] Connection header received
0929 18:37:25.493551 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:25.493559 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:25.493765 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:25.494296 (+ 531us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:37:25.494425 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:25.495583 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:25.497226 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:25.497953 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:25.498057 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:25.498209 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:25.498366 (+ 157us) server_negotiation.cc:299] Negotiation successful
0929 18:37:25.498467 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":139,"threads_started":1}
I20260929 18:37:25.498988 15963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:25.493477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.3:35143 (local address 127.0.0.1:39880)
0929 18:37:25.493940 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:25.493952 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:25.494019 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:25.494701 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:25.494705 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:25.494728 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:37:25.494864 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:25.494871 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:25.496190 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:25.496192 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:25.497514 (+ 1322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:25.497525 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:25.498602 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:25.498605 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:25.498716 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:25.498748 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 18:37:25.498801 (+ 53us) client_negotiation.cc:240] Negotiation successful
0929 18:37:25.498882 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":368,"mutex_wait_us":114,"thread_start_us":81,"threads_started":1}
I20260929 18:37:25.499310 15962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:25.493042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.1:46433 (local address 127.0.0.1:58408)
0929 18:37:25.493503 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:25.493515 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:25.493588 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:25.494805 (+ 1217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:25.494809 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:25.494826 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:37:25.494910 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:25.494919 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:25.496147 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:25.496154 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:25.497517 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:25.497525 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:25.498909 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:25.498913 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:25.499015 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:25.499051 (+ 36us) client_negotiation.cc:769] Sending connection context
0929 18:37:25.499105 (+ 54us) client_negotiation.cc:240] Negotiation successful
0929 18:37:25.499175 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":400,"mutex_wait_us":81,"thread_start_us":85,"threads_started":1}
I20260929 18:37:25.499611 15965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:25.493906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39880 (local address 127.13.100.3:35143)
0929 18:37:25.494276 (+ 370us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:25.494287 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:25.494308 (+ 21us) server_negotiation.cc:407] Connection header received
0929 18:37:25.494376 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:25.494380 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:25.494562 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:25.494708 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:37:25.494985 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:25.496100 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:25.497647 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:25.498467 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:25.499178 (+ 711us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:25.499348 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:25.499397 (+ 49us) server_negotiation.cc:299] Negotiation successful
0929 18:37:25.499482 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":106,"threads_started":1}
I20260929 18:37:25.500177 15964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:25.493575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58408 (local address 127.13.100.1:46433)
0929 18:37:25.494364 (+ 789us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:25.494389 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:25.494411 (+ 22us) server_negotiation.cc:407] Connection header received
0929 18:37:25.494485 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:25.494490 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:25.494659 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:25.495046 (+ 387us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:37:25.495145 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:25.496038 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:25.497996 (+ 1958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:25.498762 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:25.499591 (+ 829us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:25.499753 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:25.499807 (+ 54us) server_negotiation.cc:299] Negotiation successful
0929 18:37:25.500044 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":675,"thread_start_us":76,"threads_started":1}
I20260929 18:37:25.501745 15737 tablet_service.cc:1467] Processing CreateTablet for tablet ea43a068b5434c4c8b97de983b70f166 (DEFAULT_TABLE table=test-table [id=bca4adcfc13e434b952638f3713c9857]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:37:25.501775 15877 tablet_service.cc:1467] Processing CreateTablet for tablet ea43a068b5434c4c8b97de983b70f166 (DEFAULT_TABLE table=test-table [id=bca4adcfc13e434b952638f3713c9857]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:37:25.502810 15737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea43a068b5434c4c8b97de983b70f166. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:37:25.502825 15877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea43a068b5434c4c8b97de983b70f166. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:37:25.504293 15597 tablet_service.cc:1467] Processing CreateTablet for tablet ea43a068b5434c4c8b97de983b70f166 (DEFAULT_TABLE table=test-table [id=bca4adcfc13e434b952638f3713c9857]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:37:25.505188 15597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea43a068b5434c4c8b97de983b70f166. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:37:25.510322 15966 tablet_bootstrap.cc:492] T ea43a068b5434c4c8b97de983b70f166 P 601f7285b75f48539695ba51f9db205b: Bootstrap starting.
I20260929 18:37:25.512382 15966 tablet_bootstrap.cc:654] T ea43a068b5434c4c8b97de983b70f166 P 601f7285b75f48539695ba51f9db205b: Neither blocks nor log segments found. Creating new log.
I20260929 18:37:25.512496 15968 tablet_bootstrap.cc:492] T ea43a068b5434c4c8b97de983b70f166 P f5f5cca3d04940e781e1bb7af2217ae1: Bootstrap starting.
I20260929 18:37:25.512809 15967 tablet_bootstrap.cc:492] T ea43a068b5434c4c8b97de983b70f166 P 16fcd7a8ee8a47d2926ab4f3db9b48f6: Bootstrap starting.
I20260929 18:37:25.513298 15966 log.cc:824] T ea43a068b5434c4c8b97de983b70f166 P 601f7285b75f48539695ba51f9db205b: Log is configured to *not* fsync() on all Append() calls
I20260929 18:37:25.514525 15968 tablet_bootstrap.cc:654] T ea43a068b5434c4c8b97de983b70f166 P f5f5cca3d04940e781e1bb7af2217ae1: Neither blocks nor log segments found. Creating new log.
I20260929 18:37:25.514922 15967 tablet_bootstrap.cc:654] T ea43a068b5434c4c8b97de983b70f166 P 16fcd7a8ee8a47d2926ab4f3db9b48f6: Neither blocks nor log segments found. Creating new log.
I20260929 18:37:25.515306 15968 log.cc:824] T ea43a068b5434c4c8b97de983b70f166 P f5f5cca3d04940e781e1bb7af2217ae1: Log is configured to *not* fsync() on all Append() calls
I20260929 18:37:25.515432 15966 tablet_bootstrap.cc:492] T ea43a068b5434c4c8b97de983b70f166 P 601f7285b75f48539695ba51f9db205b: No bootstrap required, opened a new log
I20260929 18:37:25.515609 15967 log.cc:824] T ea43a068b5434c4c8b97de983b70f166 P 16fcd7a8ee8a47d2926ab4f3db9b48f6: Log is configured to *not* fsync() on all Append() calls
I20260929 18:37:25.515657 15966 ts_tablet_manager.cc:1397] T ea43a068b5434c4c8b97de983b70f166 P 601f7285b75f48539695ba51f9db205b: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260929 18:37:25.517130 15968 tablet_bootstrap.cc:492] T ea43a068b5434c4c8b97de983b70f166 P f5f5cca3d04940e781e1bb7af2217ae1: No bootstrap required, opened a new log
I20260929 18:37:25.517316 15968 ts_tablet_manager.cc:1397] T ea43a068b5434c4c8b97de983b70f166 P f5f5cca3d04940e781e1bb7af2217ae1: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260929 18:37:25.517554 15967 tablet_bootstrap.cc:492] T ea43a068b5434c4c8b97de983b70f166 P 16fcd7a8ee8a47d2926ab4f3db9b48f6: No bootstrap required, opened a new log
I20260929 18:37:25.517719 15967 ts_tablet_manager.cc:1397] T ea43a068b5434c4c8b97de983b70f166 P 16fcd7a8ee8a47d2926ab4f3db9b48f6: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260929 18:37:25.519104 15966 raft_consensus.cc:348] T ea43a068b5434c4c8b97de983b70f166 P 601f7285b75f48539695ba51f9db205b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "601f7285b75f48539695ba51f9db205b" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 35501 } } peers { permanent_uuid: "16fcd7a8ee8a47d2926ab4f3db9b48f6" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 35143 } } peers { permanent_uuid: "f5f5cca3d04940e781e1bb7af2217ae1" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 46433 } }
I20260929 18:37:25.519338 15966 raft_consensus.cc:374] T ea43a068b5434c4c8b97de983b70f166 P 601f7285b75f48539695ba51f9db205b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:37:25.519412 15966 raft_consensus.cc:729] T ea43a068b5434c4c8b97de983b70f166 P 601f7285b75f48539695ba51f9db205b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 601f7285b75f48539695ba51f9db205b, State: Initialized, Role: FOLLOWER
I20260929 18:37:25.519855 15966 consensus_queue.cc:260] T ea43a068b5434c4c8b97de983b70f166 P 601f7285b75f48539695ba51f9db205b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "601f7285b75f48539695ba51f9db205b" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 35501 } } peers { permanent_uuid: "16fcd7a8ee8a47d2926ab4f3db9b48f6" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 35143 } } peers { permanent_uuid: "f5f5cca3d04940e781e1bb7af2217ae1" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 46433 } }
I20260929 18:37:25.520464 15803 heartbeater.cc:503] Master 127.13.100.62:43705 was elected leader, sending a full tablet report...
I20260929 18:37:25.520812 15966 ts_tablet_manager.cc:1428] T ea43a068b5434c4c8b97de983b70f166 P 601f7285b75f48539695ba51f9db205b: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260929 18:37:25.520992 15968 raft_consensus.cc:348] T ea43a068b5434c4c8b97de983b70f166 P f5f5cca3d04940e781e1bb7af2217ae1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "601f7285b75f48539695ba51f9db205b" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 35501 } } peers { permanent_uuid: "16fcd7a8ee8a47d2926ab4f3db9b48f6" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 35143 } } peers { permanent_uuid: "f5f5cca3d04940e781e1bb7af2217ae1" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 46433 } }
I20260929 18:37:25.521250 15968 raft_consensus.cc:374] T ea43a068b5434c4c8b97de983b70f166 P f5f5cca3d04940e781e1bb7af2217ae1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:37:25.521301 15968 raft_consensus.cc:729] T ea43a068b5434c4c8b97de983b70f166 P f5f5cca3d04940e781e1bb7af2217ae1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5f5cca3d04940e781e1bb7af2217ae1, State: Initialized, Role: FOLLOWER
I20260929 18:37:25.521680 15968 consensus_queue.cc:260] T ea43a068b5434c4c8b97de983b70f166 P f5f5cca3d04940e781e1bb7af2217ae1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "601f7285b75f48539695ba51f9db205b" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 35501 } } peers { permanent_uuid: "16fcd7a8ee8a47d2926ab4f3db9b48f6" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 35143 } } peers { permanent_uuid: "f5f5cca3d04940e781e1bb7af2217ae1" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 46433 } }
I20260929 18:37:25.522245 15663 heartbeater.cc:503] Master 127.13.100.62:43705 was elected leader, sending a full tablet report...
I20260929 18:37:25.522171 15967 raft_consensus.cc:348] T ea43a068b5434c4c8b97de983b70f166 P 16fcd7a8ee8a47d2926ab4f3db9b48f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "601f7285b75f48539695ba51f9db205b" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 35501 } } peers { permanent_uuid: "16fcd7a8ee8a47d2926ab4f3db9b48f6" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 35143 } } peers { permanent_uuid: "f5f5cca3d04940e781e1bb7af2217ae1" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 46433 } }
I20260929 18:37:25.522398 15967 raft_consensus.cc:374] T ea43a068b5434c4c8b97de983b70f166 P 16fcd7a8ee8a47d2926ab4f3db9b48f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:37:25.522461 15967 raft_consensus.cc:729] T ea43a068b5434c4c8b97de983b70f166 P 16fcd7a8ee8a47d2926ab4f3db9b48f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16fcd7a8ee8a47d2926ab4f3db9b48f6, State: Initialized, Role: FOLLOWER
I20260929 18:37:25.522835 15967 consensus_queue.cc:260] T ea43a068b5434c4c8b97de983b70f166 P 16fcd7a8ee8a47d2926ab4f3db9b48f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "601f7285b75f48539695ba51f9db205b" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 35501 } } peers { permanent_uuid: "16fcd7a8ee8a47d2926ab4f3db9b48f6" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 35143 } } peers { permanent_uuid: "f5f5cca3d04940e781e1bb7af2217ae1" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 46433 } }
I20260929 18:37:25.523612 15967 ts_tablet_manager.cc:1428] T ea43a068b5434c4c8b97de983b70f166 P 16fcd7a8ee8a47d2926ab4f3db9b48f6: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 18:37:25.523612 15968 ts_tablet_manager.cc:1428] T ea43a068b5434c4c8b97de983b70f166 P f5f5cca3d04940e781e1bb7af2217ae1: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260929 18:37:25.524044 15943 heartbeater.cc:503] Master 127.13.100.62:43705 was elected leader, sending a full tablet report...
W20260929 18:37:25.687669 15944 tablet.cc:2369] T ea43a068b5434c4c8b97de983b70f166 P 16fcd7a8ee8a47d2926ab4f3db9b48f6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 18:37:25.723513 15804 tablet.cc:2369] T ea43a068b5434c4c8b97de983b70f166 P 601f7285b75f48539695ba51f9db205b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 18:37:25.764716 15664 tablet.cc:2369] T ea43a068b5434c4c8b97de983b70f166 P f5f5cca3d04940e781e1bb7af2217ae1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:37:25.851980 15972 raft_consensus.cc:482] T ea43a068b5434c4c8b97de983b70f166 P 601f7285b75f48539695ba51f9db205b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:37:25.852216 15972 raft_consensus.cc:504] T ea43a068b5434c4c8b97de983b70f166 P 601f7285b75f48539695ba51f9db205b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "601f7285b75f48539695ba51f9db205b" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 35501 } } peers { permanent_uuid: "16fcd7a8ee8a47d2926ab4f3db9b48f6" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 35143 } } peers { permanent_uuid: "f5f5cca3d04940e781e1bb7af2217ae1" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 46433 } }
I20260929 18:37:25.853348 15972 leader_election.cc:288] T ea43a068b5434c4c8b97de983b70f166 P 601f7285b75f48539695ba51f9db205b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 16fcd7a8ee8a47d2926ab4f3db9b48f6 (127.13.100.3:35143), f5f5cca3d04940e781e1bb7af2217ae1 (127.13.100.1:46433)
I20260929 18:37:25.857373 15965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:25.853730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.2:52715 (local address 127.13.100.3:35143)
0929 18:37:25.853947 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:25.853951 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:25.854062 (+ 111us) server_negotiation.cc:407] Connection header received
0929 18:37:25.854206 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:25.854212 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:25.854269 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:25.854340 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:37:25.854649 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:25.855169 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:25.856316 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:25.856717 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:25.856856 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:25.856917 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:25.857162 (+ 245us) server_negotiation.cc:299] Negotiation successful
0929 18:37:25.857236 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260929 18:37:25.857415 15964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:25.853721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.2:53157 (local address 127.13.100.1:46433)
0929 18:37:25.853849 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:25.853854 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:25.854152 (+ 298us) server_negotiation.cc:407] Connection header received
0929 18:37:25.854269 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:25.854271 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:25.854313 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:25.854410 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:37:25.854679 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:25.855166 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:25.856323 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:25.856716 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:25.856856 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:25.856923 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:25.857218 (+ 295us) server_negotiation.cc:299] Negotiation successful
0929 18:37:25.857310 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 18:37:25.857414 15976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:25.853614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.3:35143 (local address 127.13.100.2:52715)
0929 18:37:25.853967 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:25.853984 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:25.854078 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:25.854379 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:25.854384 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:25.854403 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:37:25.854528 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:25.854535 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:25.855303 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:25.855307 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:25.856185 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:25.856194 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:25.856847 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:25.856850 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:25.856999 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:25.857060 (+ 61us) client_negotiation.cc:769] Sending connection context
0929 18:37:25.857120 (+ 60us) client_negotiation.cc:240] Negotiation successful
0929 18:37:25.857211 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"mutex_wait_us":65,"thread_start_us":75,"threads_started":1}
I20260929 18:37:25.857421 15975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:25.853614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.1:46433 (local address 127.13.100.2:53157)
0929 18:37:25.854042 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:25.854058 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:25.854184 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:25.854442 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:25.854445 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:25.854461 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:37:25.854548 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:25.854552 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:25.855317 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:25.855320 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:25.856186 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:25.856195 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:25.856847 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:25.856850 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:25.856994 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:25.857087 (+ 93us) client_negotiation.cc:769] Sending connection context
0929 18:37:25.857184 (+ 97us) client_negotiation.cc:240] Negotiation successful
0929 18:37:25.857249 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":346,"mutex_wait_us":146,"thread_start_us":68,"threads_started":1}
I20260929 18:37:25.858058 15617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea43a068b5434c4c8b97de983b70f166" candidate_uuid: "601f7285b75f48539695ba51f9db205b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5f5cca3d04940e781e1bb7af2217ae1" is_pre_election: true
I20260929 18:37:25.858060 15897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea43a068b5434c4c8b97de983b70f166" candidate_uuid: "601f7285b75f48539695ba51f9db205b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16fcd7a8ee8a47d2926ab4f3db9b48f6" is_pre_election: true
I20260929 18:37:25.858362 15897 raft_consensus.cc:2454] T ea43a068b5434c4c8b97de983b70f166 P 16fcd7a8ee8a47d2926ab4f3db9b48f6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 601f7285b75f48539695ba51f9db205b in term 0.
I20260929 18:37:25.858362 15617 raft_consensus.cc:2454] T ea43a068b5434c4c8b97de983b70f166 P f5f5cca3d04940e781e1bb7af2217ae1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 601f7285b75f48539695ba51f9db205b in term 0.
I20260929 18:37:25.858877 15688 leader_election.cc:302] T ea43a068b5434c4c8b97de983b70f166 P 601f7285b75f48539695ba51f9db205b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 16fcd7a8ee8a47d2926ab4f3db9b48f6, 601f7285b75f48539695ba51f9db205b, f5f5cca3d04940e781e1bb7af2217ae1; no voters:
I20260929 18:37:25.859138 15972 raft_consensus.cc:2781] T ea43a068b5434c4c8b97de983b70f166 P 601f7285b75f48539695ba51f9db205b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:37:25.859236 15972 raft_consensus.cc:482] T ea43a068b5434c4c8b97de983b70f166 P 601f7285b75f48539695ba51f9db205b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:37:25.859355 15972 raft_consensus.cc:3037] T ea43a068b5434c4c8b97de983b70f166 P 601f7285b75f48539695ba51f9db205b [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:37:25.860375 15972 raft_consensus.cc:504] T ea43a068b5434c4c8b97de983b70f166 P 601f7285b75f48539695ba51f9db205b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "601f7285b75f48539695ba51f9db205b" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 35501 } } peers { permanent_uuid: "16fcd7a8ee8a47d2926ab4f3db9b48f6" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 35143 } } peers { permanent_uuid: "f5f5cca3d04940e781e1bb7af2217ae1" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 46433 } }
I20260929 18:37:25.860726 15972 leader_election.cc:288] T ea43a068b5434c4c8b97de983b70f166 P 601f7285b75f48539695ba51f9db205b [CANDIDATE]: Term 1 election: Requested vote from peers 16fcd7a8ee8a47d2926ab4f3db9b48f6 (127.13.100.3:35143), f5f5cca3d04940e781e1bb7af2217ae1 (127.13.100.1:46433)
I20260929 18:37:25.861147 15897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea43a068b5434c4c8b97de983b70f166" candidate_uuid: "601f7285b75f48539695ba51f9db205b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16fcd7a8ee8a47d2926ab4f3db9b48f6"
I20260929 18:37:25.861222 15617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea43a068b5434c4c8b97de983b70f166" candidate_uuid: "601f7285b75f48539695ba51f9db205b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5f5cca3d04940e781e1bb7af2217ae1"
I20260929 18:37:25.861279 15897 raft_consensus.cc:3037] T ea43a068b5434c4c8b97de983b70f166 P 16fcd7a8ee8a47d2926ab4f3db9b48f6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:37:25.861353 15617 raft_consensus.cc:3037] T ea43a068b5434c4c8b97de983b70f166 P f5f5cca3d04940e781e1bb7af2217ae1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:37:25.862673 15617 raft_consensus.cc:2454] T ea43a068b5434c4c8b97de983b70f166 P f5f5cca3d04940e781e1bb7af2217ae1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 601f7285b75f48539695ba51f9db205b in term 1.
I20260929 18:37:25.862994 15688 leader_election.cc:302] T ea43a068b5434c4c8b97de983b70f166 P 601f7285b75f48539695ba51f9db205b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 601f7285b75f48539695ba51f9db205b, f5f5cca3d04940e781e1bb7af2217ae1; no voters:
I20260929 18:37:25.863065 15897 raft_consensus.cc:2454] T ea43a068b5434c4c8b97de983b70f166 P 16fcd7a8ee8a47d2926ab4f3db9b48f6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 601f7285b75f48539695ba51f9db205b in term 1.
I20260929 18:37:25.863180 15972 raft_consensus.cc:2781] T ea43a068b5434c4c8b97de983b70f166 P 601f7285b75f48539695ba51f9db205b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:37:25.863407 15972 raft_consensus.cc:686] T ea43a068b5434c4c8b97de983b70f166 P 601f7285b75f48539695ba51f9db205b [term 1 LEADER]: Becoming Leader. State: Replica: 601f7285b75f48539695ba51f9db205b, State: Running, Role: LEADER
I20260929 18:37:25.863739 15972 consensus_queue.cc:237] T ea43a068b5434c4c8b97de983b70f166 P 601f7285b75f48539695ba51f9db205b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "601f7285b75f48539695ba51f9db205b" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 35501 } } peers { permanent_uuid: "16fcd7a8ee8a47d2926ab4f3db9b48f6" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 35143 } } peers { permanent_uuid: "f5f5cca3d04940e781e1bb7af2217ae1" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 46433 } }
I20260929 18:37:25.866763 15458 catalog_manager.cc:5495] T ea43a068b5434c4c8b97de983b70f166 P 601f7285b75f48539695ba51f9db205b reported cstate change: term changed from 0 to 1, leader changed from <none> to 601f7285b75f48539695ba51f9db205b (127.13.100.2). New cstate: current_term: 1 leader_uuid: "601f7285b75f48539695ba51f9db205b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "601f7285b75f48539695ba51f9db205b" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 35501 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "16fcd7a8ee8a47d2926ab4f3db9b48f6" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 35143 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f5f5cca3d04940e781e1bb7af2217ae1" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 46433 } health_report { overall_health: UNKNOWN } } }
Sep 29 18:37:25 dist-test-slave-j1xs krb5kdc[15415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707044, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.62@KRBTEST.COM for kudu/127.13.100.62@KRBTEST.COM
I20260929 18:37:25.972009 15984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:25.963429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:43705 (local address 127.0.0.1:39220)
0929 18:37:25.963720 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:25.963731 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:25.963862 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:25.964119 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:25.964122 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:25.964527 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:37:25.964613 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:25.964620 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:25.965349 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:25.965356 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:25.966003 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:25.966012 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:25.966466 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:25.966469 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:25.966550 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:25.967109 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:37:25.967120 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:37:25.968671 (+ 1551us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:37:25.970634 (+ 1963us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:25.970642 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:25.970646 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:25.970861 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:25.971137 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:25.971139 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:25.971141 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:25.971213 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:25.971560 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:37:25.971565 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:37:25.971662 (+ 97us) client_negotiation.cc:769] Sending connection context
0929 18:37:25.971750 (+ 88us) client_negotiation.cc:240] Negotiation successful
0929 18:37:25.971852 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":103,"threads_started":1}
I20260929 18:37:25.972020 15509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:25.963444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39220 (local address 127.13.100.62:43705)
0929 18:37:25.963557 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:25.963582 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:25.963808 (+ 226us) server_negotiation.cc:407] Connection header received
0929 18:37:25.963965 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:25.963971 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:25.964018 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:25.964103 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:25.964731 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:25.965245 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:25.966130 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:25.966359 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:25.966472 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:25.968804 (+ 2332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:25.968808 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:25.968810 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:37:25.968834 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:25.970527 (+ 1693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:25.970968 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:25.970971 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:25.970972 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:25.971049 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:25.971293 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:25.971295 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:25.971296 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:25.971471 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:25.971556 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:25.971814 (+ 258us) server_negotiation.cc:299] Negotiation successful
0929 18:37:25.971910 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 18:37:25.976107 15458 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.13.100.62@KRBTEST.COM'} at 127.0.0.1:39220:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 18:37:25.976794 15458 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:37:25.981691 15597 tablet_service.cc:1467] Processing CreateTablet for tablet b6166ec9f77a40288c11e539ff0515ad (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=01056a3f5e6345dab6cbd64a81698cd9]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 18:37:25.981634 15737 tablet_service.cc:1467] Processing CreateTablet for tablet b6166ec9f77a40288c11e539ff0515ad (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=01056a3f5e6345dab6cbd64a81698cd9]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 18:37:25.981637 15877 tablet_service.cc:1467] Processing CreateTablet for tablet b6166ec9f77a40288c11e539ff0515ad (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=01056a3f5e6345dab6cbd64a81698cd9]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 18:37:25.982342 15597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6166ec9f77a40288c11e539ff0515ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:37:25.982342 15737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6166ec9f77a40288c11e539ff0515ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:37:25.982355 15877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6166ec9f77a40288c11e539ff0515ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:37:25.984861 15967 tablet_bootstrap.cc:492] T b6166ec9f77a40288c11e539ff0515ad P 16fcd7a8ee8a47d2926ab4f3db9b48f6: Bootstrap starting.
I20260929 18:37:25.985108 15966 tablet_bootstrap.cc:492] T b6166ec9f77a40288c11e539ff0515ad P 601f7285b75f48539695ba51f9db205b: Bootstrap starting.
I20260929 18:37:25.985852 15967 tablet_bootstrap.cc:654] T b6166ec9f77a40288c11e539ff0515ad P 16fcd7a8ee8a47d2926ab4f3db9b48f6: Neither blocks nor log segments found. Creating new log.
I20260929 18:37:25.986398 15968 tablet_bootstrap.cc:492] T b6166ec9f77a40288c11e539ff0515ad P f5f5cca3d04940e781e1bb7af2217ae1: Bootstrap starting.
I20260929 18:37:25.986471 15966 tablet_bootstrap.cc:654] T b6166ec9f77a40288c11e539ff0515ad P 601f7285b75f48539695ba51f9db205b: Neither blocks nor log segments found. Creating new log.
I20260929 18:37:25.987447 15968 tablet_bootstrap.cc:654] T b6166ec9f77a40288c11e539ff0515ad P f5f5cca3d04940e781e1bb7af2217ae1: Neither blocks nor log segments found. Creating new log.
I20260929 18:37:25.987756 15966 tablet_bootstrap.cc:492] T b6166ec9f77a40288c11e539ff0515ad P 601f7285b75f48539695ba51f9db205b: No bootstrap required, opened a new log
I20260929 18:37:25.987756 15967 tablet_bootstrap.cc:492] T b6166ec9f77a40288c11e539ff0515ad P 16fcd7a8ee8a47d2926ab4f3db9b48f6: No bootstrap required, opened a new log
I20260929 18:37:25.987828 15967 ts_tablet_manager.cc:1397] T b6166ec9f77a40288c11e539ff0515ad P 16fcd7a8ee8a47d2926ab4f3db9b48f6: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 18:37:25.987828 15966 ts_tablet_manager.cc:1397] T b6166ec9f77a40288c11e539ff0515ad P 601f7285b75f48539695ba51f9db205b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 18:37:25.988265 15967 raft_consensus.cc:348] T b6166ec9f77a40288c11e539ff0515ad P 16fcd7a8ee8a47d2926ab4f3db9b48f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5f5cca3d04940e781e1bb7af2217ae1" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 46433 } } peers { permanent_uuid: "601f7285b75f48539695ba51f9db205b" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 35501 } } peers { permanent_uuid: "16fcd7a8ee8a47d2926ab4f3db9b48f6" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 35143 } }
I20260929 18:37:25.988263 15966 raft_consensus.cc:348] T b6166ec9f77a40288c11e539ff0515ad P 601f7285b75f48539695ba51f9db205b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5f5cca3d04940e781e1bb7af2217ae1" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 46433 } } peers { permanent_uuid: "601f7285b75f48539695ba51f9db205b" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 35501 } } peers { permanent_uuid: "16fcd7a8ee8a47d2926ab4f3db9b48f6" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 35143 } }
I20260929 18:37:25.988375 15967 raft_consensus.cc:374] T b6166ec9f77a40288c11e539ff0515ad P 16fcd7a8ee8a47d2926ab4f3db9b48f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:37:25.988375 15966 raft_consensus.cc:374] T b6166ec9f77a40288c11e539ff0515ad P 601f7285b75f48539695ba51f9db205b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:37:25.988410 15966 raft_consensus.cc:729] T b6166ec9f77a40288c11e539ff0515ad P 601f7285b75f48539695ba51f9db205b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 601f7285b75f48539695ba51f9db205b, State: Initialized, Role: FOLLOWER
I20260929 18:37:25.988410 15967 raft_consensus.cc:729] T b6166ec9f77a40288c11e539ff0515ad P 16fcd7a8ee8a47d2926ab4f3db9b48f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16fcd7a8ee8a47d2926ab4f3db9b48f6, State: Initialized, Role: FOLLOWER
I20260929 18:37:25.988471 15968 tablet_bootstrap.cc:492] T b6166ec9f77a40288c11e539ff0515ad P f5f5cca3d04940e781e1bb7af2217ae1: No bootstrap required, opened a new log
I20260929 18:37:25.988561 15968 ts_tablet_manager.cc:1397] T b6166ec9f77a40288c11e539ff0515ad P f5f5cca3d04940e781e1bb7af2217ae1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260929 18:37:25.988538 15966 consensus_queue.cc:260] T b6166ec9f77a40288c11e539ff0515ad P 601f7285b75f48539695ba51f9db205b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5f5cca3d04940e781e1bb7af2217ae1" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 46433 } } peers { permanent_uuid: "601f7285b75f48539695ba51f9db205b" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 35501 } } peers { permanent_uuid: "16fcd7a8ee8a47d2926ab4f3db9b48f6" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 35143 } }
I20260929 18:37:25.988538 15967 consensus_queue.cc:260] T b6166ec9f77a40288c11e539ff0515ad P 16fcd7a8ee8a47d2926ab4f3db9b48f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5f5cca3d04940e781e1bb7af2217ae1" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 46433 } } peers { permanent_uuid: "601f7285b75f48539695ba51f9db205b" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 35501 } } peers { permanent_uuid: "16fcd7a8ee8a47d2926ab4f3db9b48f6" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 35143 } }
I20260929 18:37:25.988792 15967 ts_tablet_manager.cc:1428] T b6166ec9f77a40288c11e539ff0515ad P 16fcd7a8ee8a47d2926ab4f3db9b48f6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 18:37:25.988811 15966 ts_tablet_manager.cc:1428] T b6166ec9f77a40288c11e539ff0515ad P 601f7285b75f48539695ba51f9db205b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260929 18:37:25.988801 15972 tablet_replica.cc:439] T b6166ec9f77a40288c11e539ff0515ad P 601f7285b75f48539695ba51f9db205b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5f5cca3d04940e781e1bb7af2217ae1" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 46433 } } peers { permanent_uuid: "601f7285b75f48539695ba51f9db205b" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 35501 } } peers { permanent_uuid: "16fcd7a8ee8a47d2926ab4f3db9b48f6" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 35143 } } }
I20260929 18:37:25.988942 15972 tablet_replica.cc:442] T b6166ec9f77a40288c11e539ff0515ad P 601f7285b75f48539695ba51f9db205b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 18:37:25.989005 15968 raft_consensus.cc:348] T b6166ec9f77a40288c11e539ff0515ad P f5f5cca3d04940e781e1bb7af2217ae1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5f5cca3d04940e781e1bb7af2217ae1" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 46433 } } peers { permanent_uuid: "601f7285b75f48539695ba51f9db205b" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 35501 } } peers { permanent_uuid: "16fcd7a8ee8a47d2926ab4f3db9b48f6" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 35143 } }
I20260929 18:37:25.989095 15968 raft_consensus.cc:374] T b6166ec9f77a40288c11e539ff0515ad P f5f5cca3d04940e781e1bb7af2217ae1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:37:25.989148 15968 raft_consensus.cc:729] T b6166ec9f77a40288c11e539ff0515ad P f5f5cca3d04940e781e1bb7af2217ae1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5f5cca3d04940e781e1bb7af2217ae1, State: Initialized, Role: FOLLOWER
I20260929 18:37:25.989271 15968 consensus_queue.cc:260] T b6166ec9f77a40288c11e539ff0515ad P f5f5cca3d04940e781e1bb7af2217ae1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5f5cca3d04940e781e1bb7af2217ae1" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 46433 } } peers { permanent_uuid: "601f7285b75f48539695ba51f9db205b" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 35501 } } peers { permanent_uuid: "16fcd7a8ee8a47d2926ab4f3db9b48f6" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 35143 } }
I20260929 18:37:25.989276 15974 tablet_replica.cc:439] T b6166ec9f77a40288c11e539ff0515ad P 16fcd7a8ee8a47d2926ab4f3db9b48f6: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5f5cca3d04940e781e1bb7af2217ae1" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 46433 } } peers { permanent_uuid: "601f7285b75f48539695ba51f9db205b" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 35501 } } peers { permanent_uuid: "16fcd7a8ee8a47d2926ab4f3db9b48f6" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 35143 } } }
I20260929 18:37:25.989395 15974 tablet_replica.cc:442] T b6166ec9f77a40288c11e539ff0515ad P 16fcd7a8ee8a47d2926ab4f3db9b48f6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 18:37:25.989542 15968 ts_tablet_manager.cc:1428] T b6166ec9f77a40288c11e539ff0515ad P f5f5cca3d04940e781e1bb7af2217ae1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 18:37:25.989498 15973 tablet_replica.cc:439] T b6166ec9f77a40288c11e539ff0515ad P f5f5cca3d04940e781e1bb7af2217ae1: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5f5cca3d04940e781e1bb7af2217ae1" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 46433 } } peers { permanent_uuid: "601f7285b75f48539695ba51f9db205b" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 35501 } } peers { permanent_uuid: "16fcd7a8ee8a47d2926ab4f3db9b48f6" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 35143 } } }
I20260929 18:37:25.989616 15973 tablet_replica.cc:442] T b6166ec9f77a40288c11e539ff0515ad P f5f5cca3d04940e781e1bb7af2217ae1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 18:37:26.030591 15973 raft_consensus.cc:482] T b6166ec9f77a40288c11e539ff0515ad P f5f5cca3d04940e781e1bb7af2217ae1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:37:26.030711 15973 raft_consensus.cc:504] T b6166ec9f77a40288c11e539ff0515ad P f5f5cca3d04940e781e1bb7af2217ae1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5f5cca3d04940e781e1bb7af2217ae1" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 46433 } } peers { permanent_uuid: "601f7285b75f48539695ba51f9db205b" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 35501 } } peers { permanent_uuid: "16fcd7a8ee8a47d2926ab4f3db9b48f6" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 35143 } }
I20260929 18:37:26.031888 15973 leader_election.cc:288] T b6166ec9f77a40288c11e539ff0515ad P f5f5cca3d04940e781e1bb7af2217ae1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 601f7285b75f48539695ba51f9db205b (127.13.100.2:35501), 16fcd7a8ee8a47d2926ab4f3db9b48f6 (127.13.100.3:35143)
I20260929 18:37:26.035720 15988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:26.032060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.2:35501 (local address 127.13.100.1:57625)
0929 18:37:26.032359 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:26.032376 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:26.032502 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:26.032797 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:26.032800 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:26.032813 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:37:26.032938 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:26.032943 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:26.033662 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:26.033667 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:26.034621 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:26.034630 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:26.035290 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:26.035294 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:26.035387 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:26.035408 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 18:37:26.035503 (+ 95us) client_negotiation.cc:240] Negotiation successful
0929 18:37:26.035570 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":83,"threads_started":1}
I20260929 18:37:26.035714 15989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:26.032180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.1:57625 (local address 127.13.100.2:35501)
0929 18:37:26.032417 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:26.032433 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:26.032448 (+ 15us) server_negotiation.cc:407] Connection header received
0929 18:37:26.032631 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:26.032636 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:26.032693 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:26.032768 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:37:26.033033 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:26.033552 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:26.034739 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:26.035139 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:26.035290 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:26.035354 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:26.035516 (+ 162us) server_negotiation.cc:299] Negotiation successful
0929 18:37:26.035584 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":73,"threads_started":1}
I20260929 18:37:26.035941 15990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:26.032471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.3:35143 (local address 127.13.100.1:35065)
0929 18:37:26.032693 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:26.032706 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:26.032806 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:26.033207 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:26.033210 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:26.033225 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:37:26.033315 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:26.033320 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:26.034083 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:26.034086 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:26.034839 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:26.034848 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:26.035532 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:26.035535 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:26.035624 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:26.035651 (+ 27us) client_negotiation.cc:769] Sending connection context
0929 18:37:26.035737 (+ 86us) client_negotiation.cc:240] Negotiation successful
0929 18:37:26.035802 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":77,"threads_started":1}
I20260929 18:37:26.035925 15965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:26.032573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.1:35065 (local address 127.13.100.3:35143)
0929 18:37:26.032696 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:26.032703 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:26.033026 (+ 323us) server_negotiation.cc:407] Connection header received
0929 18:37:26.033072 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:26.033074 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:26.033109 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:26.033186 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:37:26.033417 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:26.033948 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:26.034976 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:26.035412 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:26.035541 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:26.035587 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:26.035759 (+ 172us) server_negotiation.cc:299] Negotiation successful
0929 18:37:26.035819 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 18:37:26.036320 15757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6166ec9f77a40288c11e539ff0515ad" candidate_uuid: "f5f5cca3d04940e781e1bb7af2217ae1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "601f7285b75f48539695ba51f9db205b" is_pre_election: true
I20260929 18:37:26.036340 15897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6166ec9f77a40288c11e539ff0515ad" candidate_uuid: "f5f5cca3d04940e781e1bb7af2217ae1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16fcd7a8ee8a47d2926ab4f3db9b48f6" is_pre_election: true
I20260929 18:37:26.036494 15897 raft_consensus.cc:2454] T b6166ec9f77a40288c11e539ff0515ad P 16fcd7a8ee8a47d2926ab4f3db9b48f6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5f5cca3d04940e781e1bb7af2217ae1 in term 0.
I20260929 18:37:26.036579 15757 raft_consensus.cc:2454] T b6166ec9f77a40288c11e539ff0515ad P 601f7285b75f48539695ba51f9db205b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5f5cca3d04940e781e1bb7af2217ae1 in term 0.
I20260929 18:37:26.036862 15549 leader_election.cc:302] T b6166ec9f77a40288c11e539ff0515ad P f5f5cca3d04940e781e1bb7af2217ae1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16fcd7a8ee8a47d2926ab4f3db9b48f6, f5f5cca3d04940e781e1bb7af2217ae1; no voters:
I20260929 18:37:26.037154 15973 raft_consensus.cc:2781] T b6166ec9f77a40288c11e539ff0515ad P f5f5cca3d04940e781e1bb7af2217ae1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:37:26.037209 15973 raft_consensus.cc:482] T b6166ec9f77a40288c11e539ff0515ad P f5f5cca3d04940e781e1bb7af2217ae1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:37:26.037236 15973 raft_consensus.cc:3037] T b6166ec9f77a40288c11e539ff0515ad P f5f5cca3d04940e781e1bb7af2217ae1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:37:26.038260 15973 raft_consensus.cc:504] T b6166ec9f77a40288c11e539ff0515ad P f5f5cca3d04940e781e1bb7af2217ae1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5f5cca3d04940e781e1bb7af2217ae1" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 46433 } } peers { permanent_uuid: "601f7285b75f48539695ba51f9db205b" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 35501 } } peers { permanent_uuid: "16fcd7a8ee8a47d2926ab4f3db9b48f6" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 35143 } }
I20260929 18:37:26.038582 15973 leader_election.cc:288] T b6166ec9f77a40288c11e539ff0515ad P f5f5cca3d04940e781e1bb7af2217ae1 [CANDIDATE]: Term 1 election: Requested vote from peers 601f7285b75f48539695ba51f9db205b (127.13.100.2:35501), 16fcd7a8ee8a47d2926ab4f3db9b48f6 (127.13.100.3:35143)
I20260929 18:37:26.038926 15757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6166ec9f77a40288c11e539ff0515ad" candidate_uuid: "f5f5cca3d04940e781e1bb7af2217ae1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "601f7285b75f48539695ba51f9db205b"
I20260929 18:37:26.039039 15757 raft_consensus.cc:3037] T b6166ec9f77a40288c11e539ff0515ad P 601f7285b75f48539695ba51f9db205b [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:37:26.039032 15897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6166ec9f77a40288c11e539ff0515ad" candidate_uuid: "f5f5cca3d04940e781e1bb7af2217ae1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16fcd7a8ee8a47d2926ab4f3db9b48f6"
I20260929 18:37:26.039161 15897 raft_consensus.cc:3037] T b6166ec9f77a40288c11e539ff0515ad P 16fcd7a8ee8a47d2926ab4f3db9b48f6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:37:26.039983 15757 raft_consensus.cc:2454] T b6166ec9f77a40288c11e539ff0515ad P 601f7285b75f48539695ba51f9db205b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5f5cca3d04940e781e1bb7af2217ae1 in term 1.
I20260929 18:37:26.040071 15897 raft_consensus.cc:2454] T b6166ec9f77a40288c11e539ff0515ad P 16fcd7a8ee8a47d2926ab4f3db9b48f6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5f5cca3d04940e781e1bb7af2217ae1 in term 1.
I20260929 18:37:26.040238 15549 leader_election.cc:302] T b6166ec9f77a40288c11e539ff0515ad P f5f5cca3d04940e781e1bb7af2217ae1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 601f7285b75f48539695ba51f9db205b, f5f5cca3d04940e781e1bb7af2217ae1; no voters:
I20260929 18:37:26.040488 15973 raft_consensus.cc:2781] T b6166ec9f77a40288c11e539ff0515ad P f5f5cca3d04940e781e1bb7af2217ae1 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:37:26.040709 15973 raft_consensus.cc:686] T b6166ec9f77a40288c11e539ff0515ad P f5f5cca3d04940e781e1bb7af2217ae1 [term 1 LEADER]: Becoming Leader. State: Replica: f5f5cca3d04940e781e1bb7af2217ae1, State: Running, Role: LEADER
I20260929 18:37:26.041064 15973 consensus_queue.cc:237] T b6166ec9f77a40288c11e539ff0515ad P f5f5cca3d04940e781e1bb7af2217ae1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5f5cca3d04940e781e1bb7af2217ae1" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 46433 } } peers { permanent_uuid: "601f7285b75f48539695ba51f9db205b" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 35501 } } peers { permanent_uuid: "16fcd7a8ee8a47d2926ab4f3db9b48f6" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 35143 } }
I20260929 18:37:26.043386 15991 tablet_replica.cc:439] T b6166ec9f77a40288c11e539ff0515ad P f5f5cca3d04940e781e1bb7af2217ae1: TxnStatusTablet state changed. Reason: New leader f5f5cca3d04940e781e1bb7af2217ae1. Latest consensus state: current_term: 1 leader_uuid: "f5f5cca3d04940e781e1bb7af2217ae1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5f5cca3d04940e781e1bb7af2217ae1" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 46433 } } peers { permanent_uuid: "601f7285b75f48539695ba51f9db205b" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 35501 } } peers { permanent_uuid: "16fcd7a8ee8a47d2926ab4f3db9b48f6" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 35143 } } }
I20260929 18:37:26.043500 15991 tablet_replica.cc:442] T b6166ec9f77a40288c11e539ff0515ad P f5f5cca3d04940e781e1bb7af2217ae1: This TxnStatusTablet replica's current role is: LEADER
I20260929 18:37:26.043749 15993 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 18:37:26.043943 15993 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 18:37:26.044488 15460 catalog_manager.cc:5495] T b6166ec9f77a40288c11e539ff0515ad P f5f5cca3d04940e781e1bb7af2217ae1 reported cstate change: term changed from 0 to 1, leader changed from <none> to f5f5cca3d04940e781e1bb7af2217ae1 (127.13.100.1). New cstate: current_term: 1 leader_uuid: "f5f5cca3d04940e781e1bb7af2217ae1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5f5cca3d04940e781e1bb7af2217ae1" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 46433 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "601f7285b75f48539695ba51f9db205b" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 35501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16fcd7a8ee8a47d2926ab4f3db9b48f6" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 35143 } health_report { overall_health: UNKNOWN } } }
I20260929 18:37:26.069625 15964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:26.065826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58412 (local address 127.13.100.1:46433)
0929 18:37:26.065954 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:26.065962 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:26.066032 (+ 70us) server_negotiation.cc:407] Connection header received
0929 18:37:26.066231 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:26.066234 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:26.066296 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:26.066436 (+ 140us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:37:26.066689 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:26.067194 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:26.068120 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:26.068360 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:26.068469 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:26.068806 (+ 337us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:37:26.069201 (+ 395us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:37:26.069325 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:26.069403 (+ 78us) server_negotiation.cc:299] Negotiation successful
0929 18:37:26.069453 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 18:37:26.069631 15984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:26.065728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.1:46433 (local address 127.0.0.1:58412)
0929 18:37:26.065950 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:26.065967 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:26.066092 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:26.066447 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:26.066450 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:26.066457 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 18:37:26.066584 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:26.066590 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:26.067302 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:26.067307 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:26.068000 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:26.068008 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:26.068470 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:26.068474 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:26.068567 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:26.068620 (+ 53us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 18:37:26.069300 (+ 680us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 18:37:26.069315 (+ 15us) client_negotiation.cc:769] Sending connection context
0929 18:37:26.069394 (+ 79us) client_negotiation.cc:240] Negotiation successful
0929 18:37:26.069474 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260929 18:37:26.075060 15897 raft_consensus.cc:1261] T b6166ec9f77a40288c11e539ff0515ad P 16fcd7a8ee8a47d2926ab4f3db9b48f6 [term 1 FOLLOWER]: Refusing update from remote peer f5f5cca3d04940e781e1bb7af2217ae1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:37:26.075059 15757 raft_consensus.cc:1261] T b6166ec9f77a40288c11e539ff0515ad P 601f7285b75f48539695ba51f9db205b [term 1 FOLLOWER]: Refusing update from remote peer f5f5cca3d04940e781e1bb7af2217ae1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:37:26.075682 15991 consensus_queue.cc:1035] T b6166ec9f77a40288c11e539ff0515ad P f5f5cca3d04940e781e1bb7af2217ae1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "601f7285b75f48539695ba51f9db205b" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 35501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:37:26.075917 15973 consensus_queue.cc:1035] T b6166ec9f77a40288c11e539ff0515ad P f5f5cca3d04940e781e1bb7af2217ae1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16fcd7a8ee8a47d2926ab4f3db9b48f6" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 35143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:37:26.079342 15972 tablet_replica.cc:439] T b6166ec9f77a40288c11e539ff0515ad P 601f7285b75f48539695ba51f9db205b: TxnStatusTablet state changed. Reason: New leader f5f5cca3d04940e781e1bb7af2217ae1. Latest consensus state: current_term: 1 leader_uuid: "f5f5cca3d04940e781e1bb7af2217ae1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5f5cca3d04940e781e1bb7af2217ae1" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 46433 } } peers { permanent_uuid: "601f7285b75f48539695ba51f9db205b" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 35501 } } peers { permanent_uuid: "16fcd7a8ee8a47d2926ab4f3db9b48f6" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 35143 } } }
I20260929 18:37:26.079365 15974 tablet_replica.cc:439] T b6166ec9f77a40288c11e539ff0515ad P 16fcd7a8ee8a47d2926ab4f3db9b48f6: TxnStatusTablet state changed. Reason: New leader f5f5cca3d04940e781e1bb7af2217ae1. Latest consensus state: current_term: 1 leader_uuid: "f5f5cca3d04940e781e1bb7af2217ae1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5f5cca3d04940e781e1bb7af2217ae1" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 46433 } } peers { permanent_uuid: "601f7285b75f48539695ba51f9db205b" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 35501 } } peers { permanent_uuid: "16fcd7a8ee8a47d2926ab4f3db9b48f6" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 35143 } } }
I20260929 18:37:26.079463 15972 tablet_replica.cc:442] T b6166ec9f77a40288c11e539ff0515ad P 601f7285b75f48539695ba51f9db205b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 18:37:26.079475 15974 tablet_replica.cc:442] T b6166ec9f77a40288c11e539ff0515ad P 16fcd7a8ee8a47d2926ab4f3db9b48f6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 18:37:26.082891 15972 tablet_replica.cc:439] T b6166ec9f77a40288c11e539ff0515ad P 601f7285b75f48539695ba51f9db205b: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f5f5cca3d04940e781e1bb7af2217ae1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5f5cca3d04940e781e1bb7af2217ae1" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 46433 } } peers { permanent_uuid: "601f7285b75f48539695ba51f9db205b" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 35501 } } peers { permanent_uuid: "16fcd7a8ee8a47d2926ab4f3db9b48f6" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 35143 } } }
I20260929 18:37:26.083006 15972 tablet_replica.cc:442] T b6166ec9f77a40288c11e539ff0515ad P 601f7285b75f48539695ba51f9db205b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 18:37:26.083259 15973 tablet_replica.cc:439] T b6166ec9f77a40288c11e539ff0515ad P f5f5cca3d04940e781e1bb7af2217ae1: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f5f5cca3d04940e781e1bb7af2217ae1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5f5cca3d04940e781e1bb7af2217ae1" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 46433 } } peers { permanent_uuid: "601f7285b75f48539695ba51f9db205b" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 35501 } } peers { permanent_uuid: "16fcd7a8ee8a47d2926ab4f3db9b48f6" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 35143 } } }
I20260929 18:37:26.083379 15973 tablet_replica.cc:442] T b6166ec9f77a40288c11e539ff0515ad P f5f5cca3d04940e781e1bb7af2217ae1: This TxnStatusTablet replica's current role is: LEADER
I20260929 18:37:26.083514 15974 tablet_replica.cc:439] T b6166ec9f77a40288c11e539ff0515ad P 16fcd7a8ee8a47d2926ab4f3db9b48f6: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f5f5cca3d04940e781e1bb7af2217ae1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5f5cca3d04940e781e1bb7af2217ae1" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 46433 } } peers { permanent_uuid: "601f7285b75f48539695ba51f9db205b" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 35501 } } peers { permanent_uuid: "16fcd7a8ee8a47d2926ab4f3db9b48f6" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 35143 } } }
I20260929 18:37:26.083564 15996 mvcc.cc:205] Tried to move back new op lower bound from 7334736060716318720 to 7334736060587016192. Current Snapshot: MvccSnapshot[applied={T|T < 7334736060716318720}]
I20260929 18:37:26.083619 15974 tablet_replica.cc:442] T b6166ec9f77a40288c11e539ff0515ad P 16fcd7a8ee8a47d2926ab4f3db9b48f6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 18:37:26.085047 15991 tablet_replica.cc:439] T b6166ec9f77a40288c11e539ff0515ad P f5f5cca3d04940e781e1bb7af2217ae1: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f5f5cca3d04940e781e1bb7af2217ae1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5f5cca3d04940e781e1bb7af2217ae1" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 46433 } } peers { permanent_uuid: "601f7285b75f48539695ba51f9db205b" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 35501 } } peers { permanent_uuid: "16fcd7a8ee8a47d2926ab4f3db9b48f6" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 35143 } } }
I20260929 18:37:26.085162 15991 tablet_replica.cc:442] T b6166ec9f77a40288c11e539ff0515ad P f5f5cca3d04940e781e1bb7af2217ae1: This TxnStatusTablet replica's current role is: LEADER
I20260929 18:37:26.085773 15997 mvcc.cc:205] Tried to move back new op lower bound from 7334736060716318720 to 7334736060587016192. Current Snapshot: MvccSnapshot[applied={T|T < 7334736060716318720 or (T in {7334736060716318720})}]
I20260929 18:37:26.095368 15989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:26.091561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41284 (local address 127.13.100.2:35501)
0929 18:37:26.091674 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:26.091678 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:26.091692 (+ 14us) server_negotiation.cc:407] Connection header received
0929 18:37:26.091766 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:26.091769 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:26.091829 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:26.091935 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:37:26.092098 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:26.092879 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:26.093755 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:26.094135 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:26.094231 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:26.094495 (+ 264us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:37:26.095001 (+ 506us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:37:26.095115 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:26.095174 (+ 59us) server_negotiation.cc:299] Negotiation successful
0929 18:37:26.095233 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260929 18:37:26.106663 16009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:26.102916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.1:46433 (local address 127.13.100.2:34931)
0929 18:37:26.103202 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:26.103214 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:26.103298 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:26.103558 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:26.103560 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:26.103571 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 18:37:26.103683 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:26.103688 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:26.104667 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:26.104670 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:26.105335 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:26.105343 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:26.105957 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:26.105960 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:26.106035 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:26.106079 (+ 44us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 18:37:26.106395 (+ 316us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 18:37:26.106400 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 18:37:26.106449 (+ 49us) client_negotiation.cc:240] Negotiation successful
0929 18:37:26.106515 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":116,"threads_started":1}
I20260929 18:37:26.106668 15964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:26.103049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.2:34931 (local address 127.13.100.1:46433)
0929 18:37:26.103175 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:26.103180 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:26.103282 (+ 102us) server_negotiation.cc:407] Connection header received
0929 18:37:26.103407 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:26.103410 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:26.103464 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:26.103546 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:37:26.103785 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:26.104555 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:26.105443 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:26.105777 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:26.105878 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:26.106234 (+ 356us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:37:26.106314 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:37:26.106384 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:26.106483 (+ 99us) server_negotiation.cc:299] Negotiation successful
0929 18:37:26.106535 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 18:37:26.114140 15897 raft_consensus.cc:1261] T ea43a068b5434c4c8b97de983b70f166 P 16fcd7a8ee8a47d2926ab4f3db9b48f6 [term 1 FOLLOWER]: Refusing update from remote peer 601f7285b75f48539695ba51f9db205b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:37:26.114251 15617 raft_consensus.cc:1261] T ea43a068b5434c4c8b97de983b70f166 P f5f5cca3d04940e781e1bb7af2217ae1 [term 1 FOLLOWER]: Refusing update from remote peer 601f7285b75f48539695ba51f9db205b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:37:26.114744 15972 consensus_queue.cc:1035] T ea43a068b5434c4c8b97de983b70f166 P 601f7285b75f48539695ba51f9db205b [LEADER]: Connected to new peer: Peer: permanent_uuid: "16fcd7a8ee8a47d2926ab4f3db9b48f6" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 35143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:37:26.114952 15977 consensus_queue.cc:1035] T ea43a068b5434c4c8b97de983b70f166 P 601f7285b75f48539695ba51f9db205b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5f5cca3d04940e781e1bb7af2217ae1" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 46433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:37:26.142035 15989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:26.134605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.1:54509 (local address 127.13.100.2:35501)
0929 18:37:26.134761 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:26.134765 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:26.138193 (+ 3428us) server_negotiation.cc:407] Connection header received
0929 18:37:26.138386 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:26.138390 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:26.138446 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:26.138540 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:37:26.138943 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:26.139424 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:26.140671 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:26.141029 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:26.141099 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:26.141474 (+ 375us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:37:26.141553 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:37:26.141596 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:26.141817 (+ 221us) server_negotiation.cc:299] Negotiation successful
0929 18:37:26.141861 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260929 18:37:26.142046 16013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:26.134563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.2:35501 (local address 127.13.100.1:54509)
0929 18:37:26.138109 (+ 3546us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:26.138126 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:26.138266 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:26.138717 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:26.138719 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:26.138730 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 18:37:26.138835 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:26.138841 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:26.139525 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:26.139528 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:26.140496 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:26.140506 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:26.141205 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:26.141208 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:26.141290 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:26.141338 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 18:37:26.141697 (+ 359us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 18:37:26.141703 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 18:37:26.141765 (+ 62us) client_negotiation.cc:240] Negotiation successful
0929 18:37:26.141843 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3423,"thread_start_us":131,"threads_started":1}
I20260929 18:37:26.166877 15459 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39216:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:37:26.167075 15459 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39216:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:37:26.169966 15459 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bb338bdc225b47fca102578584452513: Sending DeleteTablet for 3 replicas of tablet ea43a068b5434c4c8b97de983b70f166
I20260929 18:37:26.170830 15877 tablet_service.cc:1514] Processing DeleteTablet for tablet ea43a068b5434c4c8b97de983b70f166 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:37:26 UTC) from {username='kudu', principal='kudu/127.13.100.62@KRBTEST.COM'} at 127.0.0.1:39880
I20260929 18:37:26.170883 15737 tablet_service.cc:1514] Processing DeleteTablet for tablet ea43a068b5434c4c8b97de983b70f166 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:37:26 UTC) from {username='kudu', principal='kudu/127.13.100.62@KRBTEST.COM'} at 127.0.0.1:41276
I20260929 18:37:26.171355 16018 tablet_replica.cc:330] T ea43a068b5434c4c8b97de983b70f166 P 16fcd7a8ee8a47d2926ab4f3db9b48f6: stopping tablet replica
I20260929 18:37:26.171685 16018 raft_consensus.cc:2229] T ea43a068b5434c4c8b97de983b70f166 P 16fcd7a8ee8a47d2926ab4f3db9b48f6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:37:26.171914 16018 raft_consensus.cc:2258] T ea43a068b5434c4c8b97de983b70f166 P 16fcd7a8ee8a47d2926ab4f3db9b48f6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:37:26.172494 15459 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39216:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260929 18:37:26.172614 15597 tablet_service.cc:1514] Processing DeleteTablet for tablet ea43a068b5434c4c8b97de983b70f166 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:37:26 UTC) from {username='kudu', principal='kudu/127.13.100.62@KRBTEST.COM'} at 127.0.0.1:58408
W20260929 18:37:26.172902 15459 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:37:26.173372 16017 tablet_replica.cc:330] T ea43a068b5434c4c8b97de983b70f166 P 601f7285b75f48539695ba51f9db205b: stopping tablet replica
I20260929 18:37:26.173642 16019 tablet_replica.cc:330] T ea43a068b5434c4c8b97de983b70f166 P f5f5cca3d04940e781e1bb7af2217ae1: stopping tablet replica
I20260929 18:37:26.175006 16018 ts_tablet_manager.cc:1905] T ea43a068b5434c4c8b97de983b70f166 P 16fcd7a8ee8a47d2926ab4f3db9b48f6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:37:26.178098 16018 ts_tablet_manager.cc:1918] T ea43a068b5434c4c8b97de983b70f166 P 16fcd7a8ee8a47d2926ab4f3db9b48f6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 18:37:26.178248 16018 log.cc:1196] T ea43a068b5434c4c8b97de983b70f166 P 16fcd7a8ee8a47d2926ab4f3db9b48f6: Deleting WAL directory at /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707036041078-13712-0/minicluster-data/ts-2/wal/wals/ea43a068b5434c4c8b97de983b70f166
I20260929 18:37:26.178596 16018 ts_tablet_manager.cc:1939] T ea43a068b5434c4c8b97de983b70f166 P 16fcd7a8ee8a47d2926ab4f3db9b48f6: Deleting consensus metadata
I20260929 18:37:26.178988 15877 tablet_service.cc:1467] Processing CreateTablet for tablet f17ff2e6b3944d86bd425f0e4bc4cda3 (DEFAULT_TABLE table=test-table [id=1b1c2fccbfa54d5c9eee49d291141eae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:37:26.179275 15877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f17ff2e6b3944d86bd425f0e4bc4cda3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:37:26.179749 15597 tablet_service.cc:1467] Processing CreateTablet for tablet f17ff2e6b3944d86bd425f0e4bc4cda3 (DEFAULT_TABLE table=test-table [id=1b1c2fccbfa54d5c9eee49d291141eae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:37:26.180011 15597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f17ff2e6b3944d86bd425f0e4bc4cda3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:37:26.181648 15967 tablet_bootstrap.cc:492] T f17ff2e6b3944d86bd425f0e4bc4cda3 P 16fcd7a8ee8a47d2926ab4f3db9b48f6: Bootstrap starting.
I20260929 18:37:26.182260 15968 tablet_bootstrap.cc:492] T f17ff2e6b3944d86bd425f0e4bc4cda3 P f5f5cca3d04940e781e1bb7af2217ae1: Bootstrap starting.
I20260929 18:37:26.182660 15967 tablet_bootstrap.cc:654] T f17ff2e6b3944d86bd425f0e4bc4cda3 P 16fcd7a8ee8a47d2926ab4f3db9b48f6: Neither blocks nor log segments found. Creating new log.
I20260929 18:37:26.179013 15737 tablet_service.cc:1467] Processing CreateTablet for tablet f17ff2e6b3944d86bd425f0e4bc4cda3 (DEFAULT_TABLE table=test-table [id=1b1c2fccbfa54d5c9eee49d291141eae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:37:26.183574 15737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f17ff2e6b3944d86bd425f0e4bc4cda3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:37:26.183655 15968 tablet_bootstrap.cc:654] T f17ff2e6b3944d86bd425f0e4bc4cda3 P f5f5cca3d04940e781e1bb7af2217ae1: Neither blocks nor log segments found. Creating new log.
I20260929 18:37:26.184630 15445 catalog_manager.cc:4841] TS 16fcd7a8ee8a47d2926ab4f3db9b48f6 (127.13.100.3:35143): tablet ea43a068b5434c4c8b97de983b70f166 (table test-table [id=bca4adcfc13e434b952638f3713c9857]) successfully deleted
I20260929 18:37:26.185559 15967 tablet_bootstrap.cc:492] T f17ff2e6b3944d86bd425f0e4bc4cda3 P 16fcd7a8ee8a47d2926ab4f3db9b48f6: No bootstrap required, opened a new log
I20260929 18:37:26.185639 15967 ts_tablet_manager.cc:1397] T f17ff2e6b3944d86bd425f0e4bc4cda3 P 16fcd7a8ee8a47d2926ab4f3db9b48f6: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260929 18:37:26.186121 15967 raft_consensus.cc:348] T f17ff2e6b3944d86bd425f0e4bc4cda3 P 16fcd7a8ee8a47d2926ab4f3db9b48f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5f5cca3d04940e781e1bb7af2217ae1" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 46433 } } peers { permanent_uuid: "601f7285b75f48539695ba51f9db205b" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 35501 } } peers { permanent_uuid: "16fcd7a8ee8a47d2926ab4f3db9b48f6" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 35143 } }
I20260929 18:37:26.186247 15967 raft_consensus.cc:374] T f17ff2e6b3944d86bd425f0e4bc4cda3 P 16fcd7a8ee8a47d2926ab4f3db9b48f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:37:26.186285 15967 raft_consensus.cc:729] T f17ff2e6b3944d86bd425f0e4bc4cda3 P 16fcd7a8ee8a47d2926ab4f3db9b48f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16fcd7a8ee8a47d2926ab4f3db9b48f6, State: Initialized, Role: FOLLOWER
I20260929 18:37:26.186414 15967 consensus_queue.cc:260] T f17ff2e6b3944d86bd425f0e4bc4cda3 P 16fcd7a8ee8a47d2926ab4f3db9b48f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5f5cca3d04940e781e1bb7af2217ae1" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 46433 } } peers { permanent_uuid: "601f7285b75f48539695ba51f9db205b" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 35501 } } peers { permanent_uuid: "16fcd7a8ee8a47d2926ab4f3db9b48f6" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 35143 } }
I20260929 18:37:26.186698 15967 ts_tablet_manager.cc:1428] T f17ff2e6b3944d86bd425f0e4bc4cda3 P 16fcd7a8ee8a47d2926ab4f3db9b48f6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 18:37:26.187590 15966 tablet_bootstrap.cc:492] T f17ff2e6b3944d86bd425f0e4bc4cda3 P 601f7285b75f48539695ba51f9db205b: Bootstrap starting.
I20260929 18:37:26.188125 16017 raft_consensus.cc:2229] T ea43a068b5434c4c8b97de983b70f166 P 601f7285b75f48539695ba51f9db205b [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:37:26.188277 16019 raft_consensus.cc:2229] T ea43a068b5434c4c8b97de983b70f166 P f5f5cca3d04940e781e1bb7af2217ae1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:37:26.188514 16019 raft_consensus.cc:2258] T ea43a068b5434c4c8b97de983b70f166 P f5f5cca3d04940e781e1bb7af2217ae1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:37:26.188630 15966 tablet_bootstrap.cc:654] T f17ff2e6b3944d86bd425f0e4bc4cda3 P 601f7285b75f48539695ba51f9db205b: Neither blocks nor log segments found. Creating new log.
I20260929 18:37:26.190022 15966 tablet_bootstrap.cc:492] T f17ff2e6b3944d86bd425f0e4bc4cda3 P 601f7285b75f48539695ba51f9db205b: No bootstrap required, opened a new log
I20260929 18:37:26.190110 15966 ts_tablet_manager.cc:1397] T f17ff2e6b3944d86bd425f0e4bc4cda3 P 601f7285b75f48539695ba51f9db205b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 18:37:26.190416 16017 raft_consensus.cc:2258] T ea43a068b5434c4c8b97de983b70f166 P 601f7285b75f48539695ba51f9db205b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:37:26.190543 15966 raft_consensus.cc:348] T f17ff2e6b3944d86bd425f0e4bc4cda3 P 601f7285b75f48539695ba51f9db205b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5f5cca3d04940e781e1bb7af2217ae1" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 46433 } } peers { permanent_uuid: "601f7285b75f48539695ba51f9db205b" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 35501 } } peers { permanent_uuid: "16fcd7a8ee8a47d2926ab4f3db9b48f6" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 35143 } }
I20260929 18:37:26.190651 15966 raft_consensus.cc:374] T f17ff2e6b3944d86bd425f0e4bc4cda3 P 601f7285b75f48539695ba51f9db205b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:37:26.190685 15966 raft_consensus.cc:729] T f17ff2e6b3944d86bd425f0e4bc4cda3 P 601f7285b75f48539695ba51f9db205b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 601f7285b75f48539695ba51f9db205b, State: Initialized, Role: FOLLOWER
I20260929 18:37:26.190824 15966 consensus_queue.cc:260] T f17ff2e6b3944d86bd425f0e4bc4cda3 P 601f7285b75f48539695ba51f9db205b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5f5cca3d04940e781e1bb7af2217ae1" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 46433 } } peers { permanent_uuid: "601f7285b75f48539695ba51f9db205b" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 35501 } } peers { permanent_uuid: "16fcd7a8ee8a47d2926ab4f3db9b48f6" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 35143 } }
I20260929 18:37:26.191082 15966 ts_tablet_manager.cc:1428] T f17ff2e6b3944d86bd425f0e4bc4cda3 P 601f7285b75f48539695ba51f9db205b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260929 18:37:26.194732 16017 ts_tablet_manager.cc:1905] T ea43a068b5434c4c8b97de983b70f166 P 601f7285b75f48539695ba51f9db205b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:37:26.196551 15968 tablet_bootstrap.cc:492] T f17ff2e6b3944d86bd425f0e4bc4cda3 P f5f5cca3d04940e781e1bb7af2217ae1: No bootstrap required, opened a new log
I20260929 18:37:26.196653 15968 ts_tablet_manager.cc:1397] T f17ff2e6b3944d86bd425f0e4bc4cda3 P f5f5cca3d04940e781e1bb7af2217ae1: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.001s
I20260929 18:37:26.196993 16019 ts_tablet_manager.cc:1905] T ea43a068b5434c4c8b97de983b70f166 P f5f5cca3d04940e781e1bb7af2217ae1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:37:26.197201 15968 raft_consensus.cc:348] T f17ff2e6b3944d86bd425f0e4bc4cda3 P f5f5cca3d04940e781e1bb7af2217ae1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5f5cca3d04940e781e1bb7af2217ae1" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 46433 } } peers { permanent_uuid: "601f7285b75f48539695ba51f9db205b" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 35501 } } peers { permanent_uuid: "16fcd7a8ee8a47d2926ab4f3db9b48f6" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 35143 } }
I20260929 18:37:26.197347 15968 raft_consensus.cc:374] T f17ff2e6b3944d86bd425f0e4bc4cda3 P f5f5cca3d04940e781e1bb7af2217ae1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:37:26.197394 15968 raft_consensus.cc:729] T f17ff2e6b3944d86bd425f0e4bc4cda3 P f5f5cca3d04940e781e1bb7af2217ae1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5f5cca3d04940e781e1bb7af2217ae1, State: Initialized, Role: FOLLOWER
I20260929 18:37:26.197528 15968 consensus_queue.cc:260] T f17ff2e6b3944d86bd425f0e4bc4cda3 P f5f5cca3d04940e781e1bb7af2217ae1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5f5cca3d04940e781e1bb7af2217ae1" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 46433 } } peers { permanent_uuid: "601f7285b75f48539695ba51f9db205b" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 35501 } } peers { permanent_uuid: "16fcd7a8ee8a47d2926ab4f3db9b48f6" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 35143 } }
I20260929 18:37:26.197813 15968 ts_tablet_manager.cc:1428] T f17ff2e6b3944d86bd425f0e4bc4cda3 P f5f5cca3d04940e781e1bb7af2217ae1: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260929 18:37:26.202087 16019 ts_tablet_manager.cc:1918] T ea43a068b5434c4c8b97de983b70f166 P f5f5cca3d04940e781e1bb7af2217ae1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 18:37:26.202236 16019 log.cc:1196] T ea43a068b5434c4c8b97de983b70f166 P f5f5cca3d04940e781e1bb7af2217ae1: Deleting WAL directory at /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707036041078-13712-0/minicluster-data/ts-0/wal/wals/ea43a068b5434c4c8b97de983b70f166
I20260929 18:37:26.202628 16019 ts_tablet_manager.cc:1939] T ea43a068b5434c4c8b97de983b70f166 P f5f5cca3d04940e781e1bb7af2217ae1: Deleting consensus metadata
I20260929 18:37:26.203274 16017 ts_tablet_manager.cc:1918] T ea43a068b5434c4c8b97de983b70f166 P 601f7285b75f48539695ba51f9db205b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 18:37:26.203369 16017 log.cc:1196] T ea43a068b5434c4c8b97de983b70f166 P 601f7285b75f48539695ba51f9db205b: Deleting WAL directory at /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707036041078-13712-0/minicluster-data/ts-1/wal/wals/ea43a068b5434c4c8b97de983b70f166
I20260929 18:37:26.203671 16017 ts_tablet_manager.cc:1939] T ea43a068b5434c4c8b97de983b70f166 P 601f7285b75f48539695ba51f9db205b: Deleting consensus metadata
I20260929 18:37:26.204020 15444 catalog_manager.cc:4841] TS f5f5cca3d04940e781e1bb7af2217ae1 (127.13.100.1:46433): tablet ea43a068b5434c4c8b97de983b70f166 (table test-table [id=bca4adcfc13e434b952638f3713c9857]) successfully deleted
I20260929 18:37:26.204698 15445 catalog_manager.cc:4841] TS 601f7285b75f48539695ba51f9db205b (127.13.100.2:35501): tablet ea43a068b5434c4c8b97de983b70f166 (table test-table [id=bca4adcfc13e434b952638f3713c9857]) successfully deleted
I20260929 18:37:26.438746 16002 raft_consensus.cc:482] T f17ff2e6b3944d86bd425f0e4bc4cda3 P f5f5cca3d04940e781e1bb7af2217ae1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:37:26.438894 16002 raft_consensus.cc:504] T f17ff2e6b3944d86bd425f0e4bc4cda3 P f5f5cca3d04940e781e1bb7af2217ae1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5f5cca3d04940e781e1bb7af2217ae1" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 46433 } } peers { permanent_uuid: "601f7285b75f48539695ba51f9db205b" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 35501 } } peers { permanent_uuid: "16fcd7a8ee8a47d2926ab4f3db9b48f6" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 35143 } }
I20260929 18:37:26.439318 16002 leader_election.cc:288] T f17ff2e6b3944d86bd425f0e4bc4cda3 P f5f5cca3d04940e781e1bb7af2217ae1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 601f7285b75f48539695ba51f9db205b (127.13.100.2:35501), 16fcd7a8ee8a47d2926ab4f3db9b48f6 (127.13.100.3:35143)
I20260929 18:37:26.439822 15757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f17ff2e6b3944d86bd425f0e4bc4cda3" candidate_uuid: "f5f5cca3d04940e781e1bb7af2217ae1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "601f7285b75f48539695ba51f9db205b" is_pre_election: true
I20260929 18:37:26.439879 15897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f17ff2e6b3944d86bd425f0e4bc4cda3" candidate_uuid: "f5f5cca3d04940e781e1bb7af2217ae1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16fcd7a8ee8a47d2926ab4f3db9b48f6" is_pre_election: true
I20260929 18:37:26.439996 15757 raft_consensus.cc:2454] T f17ff2e6b3944d86bd425f0e4bc4cda3 P 601f7285b75f48539695ba51f9db205b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5f5cca3d04940e781e1bb7af2217ae1 in term 0.
I20260929 18:37:26.440013 15897 raft_consensus.cc:2454] T f17ff2e6b3944d86bd425f0e4bc4cda3 P 16fcd7a8ee8a47d2926ab4f3db9b48f6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5f5cca3d04940e781e1bb7af2217ae1 in term 0.
I20260929 18:37:26.440305 15549 leader_election.cc:302] T f17ff2e6b3944d86bd425f0e4bc4cda3 P f5f5cca3d04940e781e1bb7af2217ae1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16fcd7a8ee8a47d2926ab4f3db9b48f6, f5f5cca3d04940e781e1bb7af2217ae1; no voters:
I20260929 18:37:26.440517 16002 raft_consensus.cc:2781] T f17ff2e6b3944d86bd425f0e4bc4cda3 P f5f5cca3d04940e781e1bb7af2217ae1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:37:26.440577 16002 raft_consensus.cc:482] T f17ff2e6b3944d86bd425f0e4bc4cda3 P f5f5cca3d04940e781e1bb7af2217ae1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:37:26.440617 16002 raft_consensus.cc:3037] T f17ff2e6b3944d86bd425f0e4bc4cda3 P f5f5cca3d04940e781e1bb7af2217ae1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:37:26.441731 16002 raft_consensus.cc:504] T f17ff2e6b3944d86bd425f0e4bc4cda3 P f5f5cca3d04940e781e1bb7af2217ae1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5f5cca3d04940e781e1bb7af2217ae1" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 46433 } } peers { permanent_uuid: "601f7285b75f48539695ba51f9db205b" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 35501 } } peers { permanent_uuid: "16fcd7a8ee8a47d2926ab4f3db9b48f6" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 35143 } }
I20260929 18:37:26.442123 16002 leader_election.cc:288] T f17ff2e6b3944d86bd425f0e4bc4cda3 P f5f5cca3d04940e781e1bb7af2217ae1 [CANDIDATE]: Term 1 election: Requested vote from peers 601f7285b75f48539695ba51f9db205b (127.13.100.2:35501), 16fcd7a8ee8a47d2926ab4f3db9b48f6 (127.13.100.3:35143)
I20260929 18:37:26.442451 15757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f17ff2e6b3944d86bd425f0e4bc4cda3" candidate_uuid: "f5f5cca3d04940e781e1bb7af2217ae1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "601f7285b75f48539695ba51f9db205b"
I20260929 18:37:26.442529 15897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f17ff2e6b3944d86bd425f0e4bc4cda3" candidate_uuid: "f5f5cca3d04940e781e1bb7af2217ae1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16fcd7a8ee8a47d2926ab4f3db9b48f6"
I20260929 18:37:26.442597 15757 raft_consensus.cc:3037] T f17ff2e6b3944d86bd425f0e4bc4cda3 P 601f7285b75f48539695ba51f9db205b [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:37:26.442638 15897 raft_consensus.cc:3037] T f17ff2e6b3944d86bd425f0e4bc4cda3 P 16fcd7a8ee8a47d2926ab4f3db9b48f6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:37:26.443658 15757 raft_consensus.cc:2454] T f17ff2e6b3944d86bd425f0e4bc4cda3 P 601f7285b75f48539695ba51f9db205b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5f5cca3d04940e781e1bb7af2217ae1 in term 1.
I20260929 18:37:26.443755 15897 raft_consensus.cc:2454] T f17ff2e6b3944d86bd425f0e4bc4cda3 P 16fcd7a8ee8a47d2926ab4f3db9b48f6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5f5cca3d04940e781e1bb7af2217ae1 in term 1.
I20260929 18:37:26.444075 15549 leader_election.cc:302] T f17ff2e6b3944d86bd425f0e4bc4cda3 P f5f5cca3d04940e781e1bb7af2217ae1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16fcd7a8ee8a47d2926ab4f3db9b48f6, f5f5cca3d04940e781e1bb7af2217ae1; no voters:
I20260929 18:37:26.444247 16002 raft_consensus.cc:2781] T f17ff2e6b3944d86bd425f0e4bc4cda3 P f5f5cca3d04940e781e1bb7af2217ae1 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:37:26.444324 16002 raft_consensus.cc:686] T f17ff2e6b3944d86bd425f0e4bc4cda3 P f5f5cca3d04940e781e1bb7af2217ae1 [term 1 LEADER]: Becoming Leader. State: Replica: f5f5cca3d04940e781e1bb7af2217ae1, State: Running, Role: LEADER
I20260929 18:37:26.444448 16002 consensus_queue.cc:237] T f17ff2e6b3944d86bd425f0e4bc4cda3 P f5f5cca3d04940e781e1bb7af2217ae1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5f5cca3d04940e781e1bb7af2217ae1" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 46433 } } peers { permanent_uuid: "601f7285b75f48539695ba51f9db205b" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 35501 } } peers { permanent_uuid: "16fcd7a8ee8a47d2926ab4f3db9b48f6" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 35143 } }
I20260929 18:37:26.446762 15459 catalog_manager.cc:5495] T f17ff2e6b3944d86bd425f0e4bc4cda3 P f5f5cca3d04940e781e1bb7af2217ae1 reported cstate change: term changed from 0 to 1, leader changed from <none> to f5f5cca3d04940e781e1bb7af2217ae1 (127.13.100.1). New cstate: current_term: 1 leader_uuid: "f5f5cca3d04940e781e1bb7af2217ae1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5f5cca3d04940e781e1bb7af2217ae1" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 46433 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "601f7285b75f48539695ba51f9db205b" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 35501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16fcd7a8ee8a47d2926ab4f3db9b48f6" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 35143 } health_report { overall_health: UNKNOWN } } }
I20260929 18:37:26.491561 15964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:26.488497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58416 (local address 127.13.100.1:46433)
0929 18:37:26.488602 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:26.488606 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:26.488767 (+ 161us) server_negotiation.cc:407] Connection header received
0929 18:37:26.488875 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:26.488877 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:26.488919 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:26.488984 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:37:26.489249 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:26.489741 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:26.490643 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:26.490853 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:26.490944 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:26.491151 (+ 207us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:37:26.491215 (+ 64us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:37:26.491266 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:26.491343 (+ 77us) server_negotiation.cc:299] Negotiation successful
0929 18:37:26.491389 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 18:37:26.499600 15964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:26.496932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.1:49301 (local address 127.13.100.1:46433)
0929 18:37:26.497048 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:26.497052 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:26.497088 (+ 36us) server_negotiation.cc:407] Connection header received
0929 18:37:26.497169 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:26.497171 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:26.497206 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:26.497267 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:37:26.497423 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:26.497869 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:26.498770 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:26.498985 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:26.499066 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:26.499232 (+ 166us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:37:26.499284 (+ 52us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:37:26.499333 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:26.499456 (+ 123us) server_negotiation.cc:299] Negotiation successful
0929 18:37:26.499504 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":39}
I20260929 18:37:26.499678 16013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:26.496893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.1:46433 (local address 127.13.100.1:49301)
0929 18:37:26.497034 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:26.497046 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:26.497109 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:26.497286 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:26.497288 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:26.497293 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 18:37:26.497363 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:26.497366 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:26.498041 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:26.498046 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:26.498660 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:26.498668 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:26.499081 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:26.499083 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:26.499138 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:26.499161 (+ 23us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 18:37:26.499376 (+ 215us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 18:37:26.499383 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 18:37:26.499451 (+ 68us) client_negotiation.cc:240] Negotiation successful
0929 18:37:26.499549 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"mutex_wait_us":18}
I20260929 18:37:26.505426 15897 raft_consensus.cc:1261] T f17ff2e6b3944d86bd425f0e4bc4cda3 P 16fcd7a8ee8a47d2926ab4f3db9b48f6 [term 1 FOLLOWER]: Refusing update from remote peer f5f5cca3d04940e781e1bb7af2217ae1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:37:26.505457 15757 raft_consensus.cc:1261] T f17ff2e6b3944d86bd425f0e4bc4cda3 P 601f7285b75f48539695ba51f9db205b [term 1 FOLLOWER]: Refusing update from remote peer f5f5cca3d04940e781e1bb7af2217ae1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:37:26.505813 15991 consensus_queue.cc:1035] T f17ff2e6b3944d86bd425f0e4bc4cda3 P f5f5cca3d04940e781e1bb7af2217ae1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16fcd7a8ee8a47d2926ab4f3db9b48f6" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 35143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:37:26.506069 16002 consensus_queue.cc:1035] T f17ff2e6b3944d86bd425f0e4bc4cda3 P f5f5cca3d04940e781e1bb7af2217ae1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "601f7285b75f48539695ba51f9db205b" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 35501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:37:26.509253 15998 mvcc.cc:205] Tried to move back new op lower bound from 7334736062482911232 to 7334736062238310400. Current Snapshot: MvccSnapshot[applied={T|T < 7334736062482911232}]
W20260929 18:37:26.519932 15996 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260929 18:37:28.072901 15460 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39216:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:37:28.073050 15460 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39216:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:37:28.075484 15460 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bb338bdc225b47fca102578584452513: Sending DeleteTablet for 3 replicas of tablet f17ff2e6b3944d86bd425f0e4bc4cda3
I20260929 18:37:28.075978 15597 tablet_service.cc:1514] Processing DeleteTablet for tablet f17ff2e6b3944d86bd425f0e4bc4cda3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:37:28 UTC) from {username='kudu', principal='kudu/127.13.100.62@KRBTEST.COM'} at 127.0.0.1:58408
I20260929 18:37:28.076174 15737 tablet_service.cc:1514] Processing DeleteTablet for tablet f17ff2e6b3944d86bd425f0e4bc4cda3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:37:28 UTC) from {username='kudu', principal='kudu/127.13.100.62@KRBTEST.COM'} at 127.0.0.1:41276
I20260929 18:37:28.076292 16047 tablet_replica.cc:330] T f17ff2e6b3944d86bd425f0e4bc4cda3 P f5f5cca3d04940e781e1bb7af2217ae1: stopping tablet replica
I20260929 18:37:28.076411 16047 raft_consensus.cc:2229] T f17ff2e6b3944d86bd425f0e4bc4cda3 P f5f5cca3d04940e781e1bb7af2217ae1 [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:37:28.076483 15877 tablet_service.cc:1514] Processing DeleteTablet for tablet f17ff2e6b3944d86bd425f0e4bc4cda3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:37:28 UTC) from {username='kudu', principal='kudu/127.13.100.62@KRBTEST.COM'} at 127.0.0.1:39880
I20260929 18:37:28.076537 16048 tablet_replica.cc:330] T f17ff2e6b3944d86bd425f0e4bc4cda3 P 601f7285b75f48539695ba51f9db205b: stopping tablet replica
I20260929 18:37:28.076645 16048 raft_consensus.cc:2229] T f17ff2e6b3944d86bd425f0e4bc4cda3 P 601f7285b75f48539695ba51f9db205b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:37:28.076723 16047 raft_consensus.cc:2258] T f17ff2e6b3944d86bd425f0e4bc4cda3 P f5f5cca3d04940e781e1bb7af2217ae1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:37:28.076743 16048 raft_consensus.cc:2258] T f17ff2e6b3944d86bd425f0e4bc4cda3 P 601f7285b75f48539695ba51f9db205b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:37:28.077361 16047 ts_tablet_manager.cc:1905] T f17ff2e6b3944d86bd425f0e4bc4cda3 P f5f5cca3d04940e781e1bb7af2217ae1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:37:28.077472 16048 ts_tablet_manager.cc:1905] T f17ff2e6b3944d86bd425f0e4bc4cda3 P 601f7285b75f48539695ba51f9db205b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:37:28.080298 16048 ts_tablet_manager.cc:1918] T f17ff2e6b3944d86bd425f0e4bc4cda3 P 601f7285b75f48539695ba51f9db205b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 18:37:28.080298 16047 ts_tablet_manager.cc:1918] T f17ff2e6b3944d86bd425f0e4bc4cda3 P f5f5cca3d04940e781e1bb7af2217ae1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 18:37:28.080389 16048 log.cc:1196] T f17ff2e6b3944d86bd425f0e4bc4cda3 P 601f7285b75f48539695ba51f9db205b: Deleting WAL directory at /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707036041078-13712-0/minicluster-data/ts-1/wal/wals/f17ff2e6b3944d86bd425f0e4bc4cda3
I20260929 18:37:28.080389 16047 log.cc:1196] T f17ff2e6b3944d86bd425f0e4bc4cda3 P f5f5cca3d04940e781e1bb7af2217ae1: Deleting WAL directory at /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707036041078-13712-0/minicluster-data/ts-0/wal/wals/f17ff2e6b3944d86bd425f0e4bc4cda3
I20260929 18:37:28.080660 16048 ts_tablet_manager.cc:1939] T f17ff2e6b3944d86bd425f0e4bc4cda3 P 601f7285b75f48539695ba51f9db205b: Deleting consensus metadata
I20260929 18:37:28.080667 16047 ts_tablet_manager.cc:1939] T f17ff2e6b3944d86bd425f0e4bc4cda3 P f5f5cca3d04940e781e1bb7af2217ae1: Deleting consensus metadata
I20260929 18:37:28.080943 16049 tablet_replica.cc:330] T f17ff2e6b3944d86bd425f0e4bc4cda3 P 16fcd7a8ee8a47d2926ab4f3db9b48f6: stopping tablet replica
I20260929 18:37:28.081122 16049 raft_consensus.cc:2229] T f17ff2e6b3944d86bd425f0e4bc4cda3 P 16fcd7a8ee8a47d2926ab4f3db9b48f6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:37:28.081292 16049 raft_consensus.cc:2258] T f17ff2e6b3944d86bd425f0e4bc4cda3 P 16fcd7a8ee8a47d2926ab4f3db9b48f6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:37:28.081548 15444 catalog_manager.cc:4841] TS f5f5cca3d04940e781e1bb7af2217ae1 (127.13.100.1:46433): tablet f17ff2e6b3944d86bd425f0e4bc4cda3 (table test-table [id=1b1c2fccbfa54d5c9eee49d291141eae]) successfully deleted
I20260929 18:37:28.081750 15445 catalog_manager.cc:4841] TS 601f7285b75f48539695ba51f9db205b (127.13.100.2:35501): tablet f17ff2e6b3944d86bd425f0e4bc4cda3 (table test-table [id=1b1c2fccbfa54d5c9eee49d291141eae]) successfully deleted
I20260929 18:37:28.082074 16049 ts_tablet_manager.cc:1905] T f17ff2e6b3944d86bd425f0e4bc4cda3 P 16fcd7a8ee8a47d2926ab4f3db9b48f6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:37:28.084501 16049 ts_tablet_manager.cc:1918] T f17ff2e6b3944d86bd425f0e4bc4cda3 P 16fcd7a8ee8a47d2926ab4f3db9b48f6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 18:37:28.084589 16049 log.cc:1196] T f17ff2e6b3944d86bd425f0e4bc4cda3 P 16fcd7a8ee8a47d2926ab4f3db9b48f6: Deleting WAL directory at /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707036041078-13712-0/minicluster-data/ts-2/wal/wals/f17ff2e6b3944d86bd425f0e4bc4cda3
I20260929 18:37:28.084851 16049 ts_tablet_manager.cc:1939] T f17ff2e6b3944d86bd425f0e4bc4cda3 P 16fcd7a8ee8a47d2926ab4f3db9b48f6: Deleting consensus metadata
I20260929 18:37:28.085601 15445 catalog_manager.cc:4841] TS 16fcd7a8ee8a47d2926ab4f3db9b48f6 (127.13.100.3:35143): tablet f17ff2e6b3944d86bd425f0e4bc4cda3 (table test-table [id=1b1c2fccbfa54d5c9eee49d291141eae]) successfully deleted
Sep 29 18:37:28 dist-test-slave-j1xs krb5kdc[15415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707048, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 18:37:28 dist-test-slave-j1xs krb5kdc[15415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707048, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.13.100.62@KRBTEST.COM
I20260929 18:37:28.113526 16057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:28.097557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39224 (local address 127.13.100.62:43705)
0929 18:37:28.097858 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:28.097868 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:28.098007 (+ 139us) server_negotiation.cc:407] Connection header received
0929 18:37:28.098190 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:28.098194 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:28.098247 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:28.098347 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:28.099070 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:28.099806 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:28.100695 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:28.100932 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:28.101045 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:28.109387 (+ 8342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:28.109391 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:28.109394 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:37:28.109433 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:28.111246 (+ 1813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:28.111759 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:28.111764 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:28.111765 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:28.111856 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:28.112142 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:28.112145 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:28.112147 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:28.112346 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:28.112488 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:28.113187 (+ 699us) server_negotiation.cc:299] Negotiation successful
0929 18:37:28.113380 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":100,"threads_started":1}
I20260929 18:37:28.116350 15460 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39224:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:37:28.116830 15460 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:37:28.121567 15597 tablet_service.cc:1467] Processing CreateTablet for tablet 315ba8fe05124184940e9fe45acd3c3a (DEFAULT_TABLE table=test-table [id=7d13e938647f4106bd4d97b9ce35ba8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:37:28.121629 15737 tablet_service.cc:1467] Processing CreateTablet for tablet 315ba8fe05124184940e9fe45acd3c3a (DEFAULT_TABLE table=test-table [id=7d13e938647f4106bd4d97b9ce35ba8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:37:28.121629 15877 tablet_service.cc:1467] Processing CreateTablet for tablet 315ba8fe05124184940e9fe45acd3c3a (DEFAULT_TABLE table=test-table [id=7d13e938647f4106bd4d97b9ce35ba8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:37:28.121944 15877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 315ba8fe05124184940e9fe45acd3c3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:37:28.121949 15737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 315ba8fe05124184940e9fe45acd3c3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:37:28.121943 15597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 315ba8fe05124184940e9fe45acd3c3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:37:28.124357 16059 tablet_bootstrap.cc:492] T 315ba8fe05124184940e9fe45acd3c3a P 601f7285b75f48539695ba51f9db205b: Bootstrap starting.
I20260929 18:37:28.124426 16058 tablet_bootstrap.cc:492] T 315ba8fe05124184940e9fe45acd3c3a P f5f5cca3d04940e781e1bb7af2217ae1: Bootstrap starting.
I20260929 18:37:28.124603 16060 tablet_bootstrap.cc:492] T 315ba8fe05124184940e9fe45acd3c3a P 16fcd7a8ee8a47d2926ab4f3db9b48f6: Bootstrap starting.
I20260929 18:37:28.125378 16058 tablet_bootstrap.cc:654] T 315ba8fe05124184940e9fe45acd3c3a P f5f5cca3d04940e781e1bb7af2217ae1: Neither blocks nor log segments found. Creating new log.
I20260929 18:37:28.125422 16060 tablet_bootstrap.cc:654] T 315ba8fe05124184940e9fe45acd3c3a P 16fcd7a8ee8a47d2926ab4f3db9b48f6: Neither blocks nor log segments found. Creating new log.
I20260929 18:37:28.126549 16058 tablet_bootstrap.cc:492] T 315ba8fe05124184940e9fe45acd3c3a P f5f5cca3d04940e781e1bb7af2217ae1: No bootstrap required, opened a new log
I20260929 18:37:28.126641 16058 ts_tablet_manager.cc:1397] T 315ba8fe05124184940e9fe45acd3c3a P f5f5cca3d04940e781e1bb7af2217ae1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 18:37:28.127118 16058 raft_consensus.cc:348] T 315ba8fe05124184940e9fe45acd3c3a P f5f5cca3d04940e781e1bb7af2217ae1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5f5cca3d04940e781e1bb7af2217ae1" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 46433 } } peers { permanent_uuid: "601f7285b75f48539695ba51f9db205b" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 35501 } } peers { permanent_uuid: "16fcd7a8ee8a47d2926ab4f3db9b48f6" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 35143 } }
I20260929 18:37:28.127240 16058 raft_consensus.cc:374] T 315ba8fe05124184940e9fe45acd3c3a P f5f5cca3d04940e781e1bb7af2217ae1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:37:28.127285 16058 raft_consensus.cc:729] T 315ba8fe05124184940e9fe45acd3c3a P f5f5cca3d04940e781e1bb7af2217ae1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5f5cca3d04940e781e1bb7af2217ae1, State: Initialized, Role: FOLLOWER
I20260929 18:37:28.127322 16059 tablet_bootstrap.cc:654] T 315ba8fe05124184940e9fe45acd3c3a P 601f7285b75f48539695ba51f9db205b: Neither blocks nor log segments found. Creating new log.
I20260929 18:37:28.127400 16058 consensus_queue.cc:260] T 315ba8fe05124184940e9fe45acd3c3a P f5f5cca3d04940e781e1bb7af2217ae1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5f5cca3d04940e781e1bb7af2217ae1" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 46433 } } peers { permanent_uuid: "601f7285b75f48539695ba51f9db205b" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 35501 } } peers { permanent_uuid: "16fcd7a8ee8a47d2926ab4f3db9b48f6" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 35143 } }
I20260929 18:37:28.127686 16058 ts_tablet_manager.cc:1428] T 315ba8fe05124184940e9fe45acd3c3a P f5f5cca3d04940e781e1bb7af2217ae1: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260929 18:37:28.128275 16060 tablet_bootstrap.cc:492] T 315ba8fe05124184940e9fe45acd3c3a P 16fcd7a8ee8a47d2926ab4f3db9b48f6: No bootstrap required, opened a new log
I20260929 18:37:28.128364 16060 ts_tablet_manager.cc:1397] T 315ba8fe05124184940e9fe45acd3c3a P 16fcd7a8ee8a47d2926ab4f3db9b48f6: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260929 18:37:28.128525 16059 tablet_bootstrap.cc:492] T 315ba8fe05124184940e9fe45acd3c3a P 601f7285b75f48539695ba51f9db205b: No bootstrap required, opened a new log
I20260929 18:37:28.128597 16059 ts_tablet_manager.cc:1397] T 315ba8fe05124184940e9fe45acd3c3a P 601f7285b75f48539695ba51f9db205b: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260929 18:37:28.128837 16060 raft_consensus.cc:348] T 315ba8fe05124184940e9fe45acd3c3a P 16fcd7a8ee8a47d2926ab4f3db9b48f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5f5cca3d04940e781e1bb7af2217ae1" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 46433 } } peers { permanent_uuid: "601f7285b75f48539695ba51f9db205b" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 35501 } } peers { permanent_uuid: "16fcd7a8ee8a47d2926ab4f3db9b48f6" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 35143 } }
I20260929 18:37:28.128937 16060 raft_consensus.cc:374] T 315ba8fe05124184940e9fe45acd3c3a P 16fcd7a8ee8a47d2926ab4f3db9b48f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:37:28.128978 16060 raft_consensus.cc:729] T 315ba8fe05124184940e9fe45acd3c3a P 16fcd7a8ee8a47d2926ab4f3db9b48f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16fcd7a8ee8a47d2926ab4f3db9b48f6, State: Initialized, Role: FOLLOWER
I20260929 18:37:28.128999 16059 raft_consensus.cc:348] T 315ba8fe05124184940e9fe45acd3c3a P 601f7285b75f48539695ba51f9db205b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5f5cca3d04940e781e1bb7af2217ae1" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 46433 } } peers { permanent_uuid: "601f7285b75f48539695ba51f9db205b" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 35501 } } peers { permanent_uuid: "16fcd7a8ee8a47d2926ab4f3db9b48f6" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 35143 } }
I20260929 18:37:28.129103 16059 raft_consensus.cc:374] T 315ba8fe05124184940e9fe45acd3c3a P 601f7285b75f48539695ba51f9db205b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:37:28.129151 16059 raft_consensus.cc:729] T 315ba8fe05124184940e9fe45acd3c3a P 601f7285b75f48539695ba51f9db205b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 601f7285b75f48539695ba51f9db205b, State: Initialized, Role: FOLLOWER
I20260929 18:37:28.129112 16060 consensus_queue.cc:260] T 315ba8fe05124184940e9fe45acd3c3a P 16fcd7a8ee8a47d2926ab4f3db9b48f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5f5cca3d04940e781e1bb7af2217ae1" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 46433 } } peers { permanent_uuid: "601f7285b75f48539695ba51f9db205b" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 35501 } } peers { permanent_uuid: "16fcd7a8ee8a47d2926ab4f3db9b48f6" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 35143 } }
I20260929 18:37:28.129315 16059 consensus_queue.cc:260] T 315ba8fe05124184940e9fe45acd3c3a P 601f7285b75f48539695ba51f9db205b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5f5cca3d04940e781e1bb7af2217ae1" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 46433 } } peers { permanent_uuid: "601f7285b75f48539695ba51f9db205b" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 35501 } } peers { permanent_uuid: "16fcd7a8ee8a47d2926ab4f3db9b48f6" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 35143 } }
I20260929 18:37:28.129521 16060 ts_tablet_manager.cc:1428] T 315ba8fe05124184940e9fe45acd3c3a P 16fcd7a8ee8a47d2926ab4f3db9b48f6: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20260929 18:37:28.129666 16059 ts_tablet_manager.cc:1428] T 315ba8fe05124184940e9fe45acd3c3a P 601f7285b75f48539695ba51f9db205b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 18:37:28.257966 16065 raft_consensus.cc:482] T 315ba8fe05124184940e9fe45acd3c3a P 601f7285b75f48539695ba51f9db205b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:37:28.258155 16065 raft_consensus.cc:504] T 315ba8fe05124184940e9fe45acd3c3a P 601f7285b75f48539695ba51f9db205b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5f5cca3d04940e781e1bb7af2217ae1" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 46433 } } peers { permanent_uuid: "601f7285b75f48539695ba51f9db205b" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 35501 } } peers { permanent_uuid: "16fcd7a8ee8a47d2926ab4f3db9b48f6" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 35143 } }
I20260929 18:37:28.258713 16065 leader_election.cc:288] T 315ba8fe05124184940e9fe45acd3c3a P 601f7285b75f48539695ba51f9db205b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f5f5cca3d04940e781e1bb7af2217ae1 (127.13.100.1:46433), 16fcd7a8ee8a47d2926ab4f3db9b48f6 (127.13.100.3:35143)
I20260929 18:37:28.259140 15617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "315ba8fe05124184940e9fe45acd3c3a" candidate_uuid: "601f7285b75f48539695ba51f9db205b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5f5cca3d04940e781e1bb7af2217ae1" is_pre_election: true
I20260929 18:37:28.259236 15897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "315ba8fe05124184940e9fe45acd3c3a" candidate_uuid: "601f7285b75f48539695ba51f9db205b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16fcd7a8ee8a47d2926ab4f3db9b48f6" is_pre_election: true
I20260929 18:37:28.259308 15617 raft_consensus.cc:2454] T 315ba8fe05124184940e9fe45acd3c3a P f5f5cca3d04940e781e1bb7af2217ae1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 601f7285b75f48539695ba51f9db205b in term 0.
I20260929 18:37:28.259413 15897 raft_consensus.cc:2454] T 315ba8fe05124184940e9fe45acd3c3a P 16fcd7a8ee8a47d2926ab4f3db9b48f6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 601f7285b75f48539695ba51f9db205b in term 0.
I20260929 18:37:28.259649 15688 leader_election.cc:302] T 315ba8fe05124184940e9fe45acd3c3a P 601f7285b75f48539695ba51f9db205b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 601f7285b75f48539695ba51f9db205b, f5f5cca3d04940e781e1bb7af2217ae1; no voters:
I20260929 18:37:28.259806 16065 raft_consensus.cc:2781] T 315ba8fe05124184940e9fe45acd3c3a P 601f7285b75f48539695ba51f9db205b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:37:28.259864 16065 raft_consensus.cc:482] T 315ba8fe05124184940e9fe45acd3c3a P 601f7285b75f48539695ba51f9db205b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:37:28.259902 16065 raft_consensus.cc:3037] T 315ba8fe05124184940e9fe45acd3c3a P 601f7285b75f48539695ba51f9db205b [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:37:28.260893 16065 raft_consensus.cc:504] T 315ba8fe05124184940e9fe45acd3c3a P 601f7285b75f48539695ba51f9db205b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5f5cca3d04940e781e1bb7af2217ae1" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 46433 } } peers { permanent_uuid: "601f7285b75f48539695ba51f9db205b" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 35501 } } peers { permanent_uuid: "16fcd7a8ee8a47d2926ab4f3db9b48f6" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 35143 } }
I20260929 18:37:28.261224 16065 leader_election.cc:288] T 315ba8fe05124184940e9fe45acd3c3a P 601f7285b75f48539695ba51f9db205b [CANDIDATE]: Term 1 election: Requested vote from peers f5f5cca3d04940e781e1bb7af2217ae1 (127.13.100.1:46433), 16fcd7a8ee8a47d2926ab4f3db9b48f6 (127.13.100.3:35143)
I20260929 18:37:28.261524 15617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "315ba8fe05124184940e9fe45acd3c3a" candidate_uuid: "601f7285b75f48539695ba51f9db205b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5f5cca3d04940e781e1bb7af2217ae1"
I20260929 18:37:28.261610 15897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "315ba8fe05124184940e9fe45acd3c3a" candidate_uuid: "601f7285b75f48539695ba51f9db205b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16fcd7a8ee8a47d2926ab4f3db9b48f6"
I20260929 18:37:28.261658 15617 raft_consensus.cc:3037] T 315ba8fe05124184940e9fe45acd3c3a P f5f5cca3d04940e781e1bb7af2217ae1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:37:28.261739 15897 raft_consensus.cc:3037] T 315ba8fe05124184940e9fe45acd3c3a P 16fcd7a8ee8a47d2926ab4f3db9b48f6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:37:28.262844 15897 raft_consensus.cc:2454] T 315ba8fe05124184940e9fe45acd3c3a P 16fcd7a8ee8a47d2926ab4f3db9b48f6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 601f7285b75f48539695ba51f9db205b in term 1.
I20260929 18:37:28.262844 15617 raft_consensus.cc:2454] T 315ba8fe05124184940e9fe45acd3c3a P f5f5cca3d04940e781e1bb7af2217ae1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 601f7285b75f48539695ba51f9db205b in term 1.
I20260929 18:37:28.263144 15688 leader_election.cc:302] T 315ba8fe05124184940e9fe45acd3c3a P 601f7285b75f48539695ba51f9db205b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 601f7285b75f48539695ba51f9db205b, f5f5cca3d04940e781e1bb7af2217ae1; no voters:
I20260929 18:37:28.263324 16065 raft_consensus.cc:2781] T 315ba8fe05124184940e9fe45acd3c3a P 601f7285b75f48539695ba51f9db205b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:37:28.263587 16065 raft_consensus.cc:686] T 315ba8fe05124184940e9fe45acd3c3a P 601f7285b75f48539695ba51f9db205b [term 1 LEADER]: Becoming Leader. State: Replica: 601f7285b75f48539695ba51f9db205b, State: Running, Role: LEADER
I20260929 18:37:28.263821 16065 consensus_queue.cc:237] T 315ba8fe05124184940e9fe45acd3c3a P 601f7285b75f48539695ba51f9db205b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5f5cca3d04940e781e1bb7af2217ae1" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 46433 } } peers { permanent_uuid: "601f7285b75f48539695ba51f9db205b" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 35501 } } peers { permanent_uuid: "16fcd7a8ee8a47d2926ab4f3db9b48f6" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 35143 } }
I20260929 18:37:28.265594 15458 catalog_manager.cc:5495] T 315ba8fe05124184940e9fe45acd3c3a P 601f7285b75f48539695ba51f9db205b reported cstate change: term changed from 0 to 1, leader changed from <none> to 601f7285b75f48539695ba51f9db205b (127.13.100.2). New cstate: current_term: 1 leader_uuid: "601f7285b75f48539695ba51f9db205b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5f5cca3d04940e781e1bb7af2217ae1" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 46433 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "601f7285b75f48539695ba51f9db205b" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 35501 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "16fcd7a8ee8a47d2926ab4f3db9b48f6" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 35143 } health_report { overall_health: UNKNOWN } } }
I20260929 18:37:28.287683 16069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:28.284168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41294 (local address 127.13.100.2:35501)
0929 18:37:28.284455 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:28.284463 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:28.284475 (+ 12us) server_negotiation.cc:407] Connection header received
0929 18:37:28.284513 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:28.284515 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:28.284555 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:28.284634 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:37:28.284850 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:28.285382 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:28.286478 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:28.286706 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:28.286797 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:28.287103 (+ 306us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:37:28.287188 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:37:28.287260 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:28.287408 (+ 148us) server_negotiation.cc:299] Negotiation successful
0929 18:37:28.287459 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":132,"threads_started":1}
I20260929 18:37:28.294667 15897 raft_consensus.cc:1261] T 315ba8fe05124184940e9fe45acd3c3a P 16fcd7a8ee8a47d2926ab4f3db9b48f6 [term 1 FOLLOWER]: Refusing update from remote peer 601f7285b75f48539695ba51f9db205b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:37:28.294709 15617 raft_consensus.cc:1261] T 315ba8fe05124184940e9fe45acd3c3a P f5f5cca3d04940e781e1bb7af2217ae1 [term 1 FOLLOWER]: Refusing update from remote peer 601f7285b75f48539695ba51f9db205b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:37:28.295111 16066 consensus_queue.cc:1035] T 315ba8fe05124184940e9fe45acd3c3a P 601f7285b75f48539695ba51f9db205b [LEADER]: Connected to new peer: Peer: permanent_uuid: "16fcd7a8ee8a47d2926ab4f3db9b48f6" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 35143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:37:28.295275 16065 consensus_queue.cc:1035] T 315ba8fe05124184940e9fe45acd3c3a P 601f7285b75f48539695ba51f9db205b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5f5cca3d04940e781e1bb7af2217ae1" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 46433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:37:28.321628 15460 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39224:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:37:28.321816 15460 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39224:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:37:28.323740 15460 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bb338bdc225b47fca102578584452513: Sending DeleteTablet for 3 replicas of tablet 315ba8fe05124184940e9fe45acd3c3a
I20260929 18:37:28.324239 15597 tablet_service.cc:1514] Processing DeleteTablet for tablet 315ba8fe05124184940e9fe45acd3c3a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:37:28 UTC) from {username='kudu', principal='kudu/127.13.100.62@KRBTEST.COM'} at 127.0.0.1:58408
I20260929 18:37:28.324368 15737 tablet_service.cc:1514] Processing DeleteTablet for tablet 315ba8fe05124184940e9fe45acd3c3a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:37:28 UTC) from {username='kudu', principal='kudu/127.13.100.62@KRBTEST.COM'} at 127.0.0.1:41276
I20260929 18:37:28.324404 15877 tablet_service.cc:1514] Processing DeleteTablet for tablet 315ba8fe05124184940e9fe45acd3c3a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:37:28 UTC) from {username='kudu', principal='kudu/127.13.100.62@KRBTEST.COM'} at 127.0.0.1:39880
I20260929 18:37:28.324569 16048 tablet_replica.cc:330] T 315ba8fe05124184940e9fe45acd3c3a P 601f7285b75f48539695ba51f9db205b: stopping tablet replica
I20260929 18:37:28.324594 16047 tablet_replica.cc:330] T 315ba8fe05124184940e9fe45acd3c3a P f5f5cca3d04940e781e1bb7af2217ae1: stopping tablet replica
I20260929 18:37:28.324586 16049 tablet_replica.cc:330] T 315ba8fe05124184940e9fe45acd3c3a P 16fcd7a8ee8a47d2926ab4f3db9b48f6: stopping tablet replica
I20260929 18:37:28.324721 16049 raft_consensus.cc:2229] T 315ba8fe05124184940e9fe45acd3c3a P 16fcd7a8ee8a47d2926ab4f3db9b48f6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:37:28.324829 16049 raft_consensus.cc:2258] T 315ba8fe05124184940e9fe45acd3c3a P 16fcd7a8ee8a47d2926ab4f3db9b48f6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:37:28.324847 16048 raft_consensus.cc:2229] T 315ba8fe05124184940e9fe45acd3c3a P 601f7285b75f48539695ba51f9db205b [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:37:28.324851 16047 raft_consensus.cc:2229] T 315ba8fe05124184940e9fe45acd3c3a P f5f5cca3d04940e781e1bb7af2217ae1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:37:28.324998 16047 raft_consensus.cc:2258] T 315ba8fe05124184940e9fe45acd3c3a P f5f5cca3d04940e781e1bb7af2217ae1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:37:28.325086 16048 raft_consensus.cc:2258] T 315ba8fe05124184940e9fe45acd3c3a P 601f7285b75f48539695ba51f9db205b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:37:28.325745 16048 ts_tablet_manager.cc:1905] T 315ba8fe05124184940e9fe45acd3c3a P 601f7285b75f48539695ba51f9db205b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:37:28.325937 15460 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39224:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260929 18:37:28.326210 16047 ts_tablet_manager.cc:1905] T 315ba8fe05124184940e9fe45acd3c3a P f5f5cca3d04940e781e1bb7af2217ae1: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260929 18:37:28.326318 15460 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:37:28.328398 16048 ts_tablet_manager.cc:1918] T 315ba8fe05124184940e9fe45acd3c3a P 601f7285b75f48539695ba51f9db205b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 18:37:28.328462 16048 log.cc:1196] T 315ba8fe05124184940e9fe45acd3c3a P 601f7285b75f48539695ba51f9db205b: Deleting WAL directory at /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707036041078-13712-0/minicluster-data/ts-1/wal/wals/315ba8fe05124184940e9fe45acd3c3a
I20260929 18:37:28.328732 16048 ts_tablet_manager.cc:1939] T 315ba8fe05124184940e9fe45acd3c3a P 601f7285b75f48539695ba51f9db205b: Deleting consensus metadata
I20260929 18:37:28.328848 16047 ts_tablet_manager.cc:1918] T 315ba8fe05124184940e9fe45acd3c3a P f5f5cca3d04940e781e1bb7af2217ae1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 18:37:28.328922 16047 log.cc:1196] T 315ba8fe05124184940e9fe45acd3c3a P f5f5cca3d04940e781e1bb7af2217ae1: Deleting WAL directory at /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707036041078-13712-0/minicluster-data/ts-0/wal/wals/315ba8fe05124184940e9fe45acd3c3a
I20260929 18:37:28.329190 16047 ts_tablet_manager.cc:1939] T 315ba8fe05124184940e9fe45acd3c3a P f5f5cca3d04940e781e1bb7af2217ae1: Deleting consensus metadata
I20260929 18:37:28.329375 15445 catalog_manager.cc:4841] TS 601f7285b75f48539695ba51f9db205b (127.13.100.2:35501): tablet 315ba8fe05124184940e9fe45acd3c3a (table test-table [id=7d13e938647f4106bd4d97b9ce35ba8a]) successfully deleted
I20260929 18:37:28.329816 15444 catalog_manager.cc:4841] TS f5f5cca3d04940e781e1bb7af2217ae1 (127.13.100.1:46433): tablet 315ba8fe05124184940e9fe45acd3c3a (table test-table [id=7d13e938647f4106bd4d97b9ce35ba8a]) successfully deleted
I20260929 18:37:28.330407 16049 ts_tablet_manager.cc:1905] T 315ba8fe05124184940e9fe45acd3c3a P 16fcd7a8ee8a47d2926ab4f3db9b48f6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:37:28.331476 15597 tablet_service.cc:1467] Processing CreateTablet for tablet d238a84052ad401cb969133851c14120 (DEFAULT_TABLE table=test-table [id=996f545c5c194ded81171c11a3f0a8ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:37:28.331496 15737 tablet_service.cc:1467] Processing CreateTablet for tablet d238a84052ad401cb969133851c14120 (DEFAULT_TABLE table=test-table [id=996f545c5c194ded81171c11a3f0a8ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:37:28.331506 15877 tablet_service.cc:1467] Processing CreateTablet for tablet d238a84052ad401cb969133851c14120 (DEFAULT_TABLE table=test-table [id=996f545c5c194ded81171c11a3f0a8ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:37:28.331732 15597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d238a84052ad401cb969133851c14120. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:37:28.331736 15877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d238a84052ad401cb969133851c14120. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:37:28.331736 15737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d238a84052ad401cb969133851c14120. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:37:28.332770 16049 ts_tablet_manager.cc:1918] T 315ba8fe05124184940e9fe45acd3c3a P 16fcd7a8ee8a47d2926ab4f3db9b48f6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 18:37:28.332837 16049 log.cc:1196] T 315ba8fe05124184940e9fe45acd3c3a P 16fcd7a8ee8a47d2926ab4f3db9b48f6: Deleting WAL directory at /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707036041078-13712-0/minicluster-data/ts-2/wal/wals/315ba8fe05124184940e9fe45acd3c3a
I20260929 18:37:28.333097 16049 ts_tablet_manager.cc:1939] T 315ba8fe05124184940e9fe45acd3c3a P 16fcd7a8ee8a47d2926ab4f3db9b48f6: Deleting consensus metadata
I20260929 18:37:28.333714 15445 catalog_manager.cc:4841] TS 16fcd7a8ee8a47d2926ab4f3db9b48f6 (127.13.100.3:35143): tablet 315ba8fe05124184940e9fe45acd3c3a (table test-table [id=7d13e938647f4106bd4d97b9ce35ba8a]) successfully deleted
I20260929 18:37:28.333841 16058 tablet_bootstrap.cc:492] T d238a84052ad401cb969133851c14120 P f5f5cca3d04940e781e1bb7af2217ae1: Bootstrap starting.
I20260929 18:37:28.333937 16059 tablet_bootstrap.cc:492] T d238a84052ad401cb969133851c14120 P 601f7285b75f48539695ba51f9db205b: Bootstrap starting.
I20260929 18:37:28.333961 16060 tablet_bootstrap.cc:492] T d238a84052ad401cb969133851c14120 P 16fcd7a8ee8a47d2926ab4f3db9b48f6: Bootstrap starting.
I20260929 18:37:28.334678 16059 tablet_bootstrap.cc:654] T d238a84052ad401cb969133851c14120 P 601f7285b75f48539695ba51f9db205b: Neither blocks nor log segments found. Creating new log.
I20260929 18:37:28.334678 16058 tablet_bootstrap.cc:654] T d238a84052ad401cb969133851c14120 P f5f5cca3d04940e781e1bb7af2217ae1: Neither blocks nor log segments found. Creating new log.
I20260929 18:37:28.334694 16060 tablet_bootstrap.cc:654] T d238a84052ad401cb969133851c14120 P 16fcd7a8ee8a47d2926ab4f3db9b48f6: Neither blocks nor log segments found. Creating new log.
I20260929 18:37:28.335769 16058 tablet_bootstrap.cc:492] T d238a84052ad401cb969133851c14120 P f5f5cca3d04940e781e1bb7af2217ae1: No bootstrap required, opened a new log
I20260929 18:37:28.335850 16058 ts_tablet_manager.cc:1397] T d238a84052ad401cb969133851c14120 P f5f5cca3d04940e781e1bb7af2217ae1: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 18:37:28.336054 16059 tablet_bootstrap.cc:492] T d238a84052ad401cb969133851c14120 P 601f7285b75f48539695ba51f9db205b: No bootstrap required, opened a new log
I20260929 18:37:28.336150 16059 ts_tablet_manager.cc:1397] T d238a84052ad401cb969133851c14120 P 601f7285b75f48539695ba51f9db205b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 18:37:28.336297 16058 raft_consensus.cc:348] T d238a84052ad401cb969133851c14120 P f5f5cca3d04940e781e1bb7af2217ae1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5f5cca3d04940e781e1bb7af2217ae1" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 46433 } } peers { permanent_uuid: "601f7285b75f48539695ba51f9db205b" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 35501 } } peers { permanent_uuid: "16fcd7a8ee8a47d2926ab4f3db9b48f6" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 35143 } }
I20260929 18:37:28.336410 16058 raft_consensus.cc:374] T d238a84052ad401cb969133851c14120 P f5f5cca3d04940e781e1bb7af2217ae1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:37:28.336437 16058 raft_consensus.cc:729] T d238a84052ad401cb969133851c14120 P f5f5cca3d04940e781e1bb7af2217ae1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5f5cca3d04940e781e1bb7af2217ae1, State: Initialized, Role: FOLLOWER
I20260929 18:37:28.336524 16059 raft_consensus.cc:348] T d238a84052ad401cb969133851c14120 P 601f7285b75f48539695ba51f9db205b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5f5cca3d04940e781e1bb7af2217ae1" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 46433 } } peers { permanent_uuid: "601f7285b75f48539695ba51f9db205b" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 35501 } } peers { permanent_uuid: "16fcd7a8ee8a47d2926ab4f3db9b48f6" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 35143 } }
I20260929 18:37:28.336642 16059 raft_consensus.cc:374] T d238a84052ad401cb969133851c14120 P 601f7285b75f48539695ba51f9db205b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:37:28.336586 16058 consensus_queue.cc:260] T d238a84052ad401cb969133851c14120 P f5f5cca3d04940e781e1bb7af2217ae1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5f5cca3d04940e781e1bb7af2217ae1" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 46433 } } peers { permanent_uuid: "601f7285b75f48539695ba51f9db205b" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 35501 } } peers { permanent_uuid: "16fcd7a8ee8a47d2926ab4f3db9b48f6" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 35143 } }
I20260929 18:37:28.336679 16059 raft_consensus.cc:729] T d238a84052ad401cb969133851c14120 P 601f7285b75f48539695ba51f9db205b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 601f7285b75f48539695ba51f9db205b, State: Initialized, Role: FOLLOWER
I20260929 18:37:28.336789 16059 consensus_queue.cc:260] T d238a84052ad401cb969133851c14120 P 601f7285b75f48539695ba51f9db205b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5f5cca3d04940e781e1bb7af2217ae1" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 46433 } } peers { permanent_uuid: "601f7285b75f48539695ba51f9db205b" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 35501 } } peers { permanent_uuid: "16fcd7a8ee8a47d2926ab4f3db9b48f6" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 35143 } }
I20260929 18:37:28.336907 16058 ts_tablet_manager.cc:1428] T d238a84052ad401cb969133851c14120 P f5f5cca3d04940e781e1bb7af2217ae1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 18:37:28.337081 16059 ts_tablet_manager.cc:1428] T d238a84052ad401cb969133851c14120 P 601f7285b75f48539695ba51f9db205b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260929 18:37:28.337452 16060 tablet_bootstrap.cc:492] T d238a84052ad401cb969133851c14120 P 16fcd7a8ee8a47d2926ab4f3db9b48f6: No bootstrap required, opened a new log
I20260929 18:37:28.337544 16060 ts_tablet_manager.cc:1397] T d238a84052ad401cb969133851c14120 P 16fcd7a8ee8a47d2926ab4f3db9b48f6: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260929 18:37:28.338029 16060 raft_consensus.cc:348] T d238a84052ad401cb969133851c14120 P 16fcd7a8ee8a47d2926ab4f3db9b48f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5f5cca3d04940e781e1bb7af2217ae1" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 46433 } } peers { permanent_uuid: "601f7285b75f48539695ba51f9db205b" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 35501 } } peers { permanent_uuid: "16fcd7a8ee8a47d2926ab4f3db9b48f6" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 35143 } }
I20260929 18:37:28.338157 16060 raft_consensus.cc:374] T d238a84052ad401cb969133851c14120 P 16fcd7a8ee8a47d2926ab4f3db9b48f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:37:28.338212 16060 raft_consensus.cc:729] T d238a84052ad401cb969133851c14120 P 16fcd7a8ee8a47d2926ab4f3db9b48f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16fcd7a8ee8a47d2926ab4f3db9b48f6, State: Initialized, Role: FOLLOWER
I20260929 18:37:28.338372 16060 consensus_queue.cc:260] T d238a84052ad401cb969133851c14120 P 16fcd7a8ee8a47d2926ab4f3db9b48f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5f5cca3d04940e781e1bb7af2217ae1" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 46433 } } peers { permanent_uuid: "601f7285b75f48539695ba51f9db205b" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 35501 } } peers { permanent_uuid: "16fcd7a8ee8a47d2926ab4f3db9b48f6" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 35143 } }
I20260929 18:37:28.338690 16060 ts_tablet_manager.cc:1428] T d238a84052ad401cb969133851c14120 P 16fcd7a8ee8a47d2926ab4f3db9b48f6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 18:37:28.715204 16030 raft_consensus.cc:482] T d238a84052ad401cb969133851c14120 P f5f5cca3d04940e781e1bb7af2217ae1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:37:28.715342 16030 raft_consensus.cc:504] T d238a84052ad401cb969133851c14120 P f5f5cca3d04940e781e1bb7af2217ae1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5f5cca3d04940e781e1bb7af2217ae1" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 46433 } } peers { permanent_uuid: "601f7285b75f48539695ba51f9db205b" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 35501 } } peers { permanent_uuid: "16fcd7a8ee8a47d2926ab4f3db9b48f6" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 35143 } }
I20260929 18:37:28.715863 16030 leader_election.cc:288] T d238a84052ad401cb969133851c14120 P f5f5cca3d04940e781e1bb7af2217ae1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 601f7285b75f48539695ba51f9db205b (127.13.100.2:35501), 16fcd7a8ee8a47d2926ab4f3db9b48f6 (127.13.100.3:35143)
I20260929 18:37:28.716419 15757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d238a84052ad401cb969133851c14120" candidate_uuid: "f5f5cca3d04940e781e1bb7af2217ae1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "601f7285b75f48539695ba51f9db205b" is_pre_election: true
I20260929 18:37:28.716475 15896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d238a84052ad401cb969133851c14120" candidate_uuid: "f5f5cca3d04940e781e1bb7af2217ae1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16fcd7a8ee8a47d2926ab4f3db9b48f6" is_pre_election: true
I20260929 18:37:28.716580 15757 raft_consensus.cc:2454] T d238a84052ad401cb969133851c14120 P 601f7285b75f48539695ba51f9db205b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5f5cca3d04940e781e1bb7af2217ae1 in term 0.
I20260929 18:37:28.716639 15896 raft_consensus.cc:2454] T d238a84052ad401cb969133851c14120 P 16fcd7a8ee8a47d2926ab4f3db9b48f6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5f5cca3d04940e781e1bb7af2217ae1 in term 0.
I20260929 18:37:28.716935 15549 leader_election.cc:302] T d238a84052ad401cb969133851c14120 P f5f5cca3d04940e781e1bb7af2217ae1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 601f7285b75f48539695ba51f9db205b, f5f5cca3d04940e781e1bb7af2217ae1; no voters:
I20260929 18:37:28.717090 16030 raft_consensus.cc:2781] T d238a84052ad401cb969133851c14120 P f5f5cca3d04940e781e1bb7af2217ae1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:37:28.717147 16030 raft_consensus.cc:482] T d238a84052ad401cb969133851c14120 P f5f5cca3d04940e781e1bb7af2217ae1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:37:28.717175 16030 raft_consensus.cc:3037] T d238a84052ad401cb969133851c14120 P f5f5cca3d04940e781e1bb7af2217ae1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:37:28.718257 16030 raft_consensus.cc:504] T d238a84052ad401cb969133851c14120 P f5f5cca3d04940e781e1bb7af2217ae1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5f5cca3d04940e781e1bb7af2217ae1" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 46433 } } peers { permanent_uuid: "601f7285b75f48539695ba51f9db205b" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 35501 } } peers { permanent_uuid: "16fcd7a8ee8a47d2926ab4f3db9b48f6" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 35143 } }
I20260929 18:37:28.718631 16030 leader_election.cc:288] T d238a84052ad401cb969133851c14120 P f5f5cca3d04940e781e1bb7af2217ae1 [CANDIDATE]: Term 1 election: Requested vote from peers 601f7285b75f48539695ba51f9db205b (127.13.100.2:35501), 16fcd7a8ee8a47d2926ab4f3db9b48f6 (127.13.100.3:35143)
I20260929 18:37:28.718968 15757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d238a84052ad401cb969133851c14120" candidate_uuid: "f5f5cca3d04940e781e1bb7af2217ae1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "601f7285b75f48539695ba51f9db205b"
I20260929 18:37:28.719095 15757 raft_consensus.cc:3037] T d238a84052ad401cb969133851c14120 P 601f7285b75f48539695ba51f9db205b [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:37:28.719089 15896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d238a84052ad401cb969133851c14120" candidate_uuid: "f5f5cca3d04940e781e1bb7af2217ae1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16fcd7a8ee8a47d2926ab4f3db9b48f6"
I20260929 18:37:28.719214 15896 raft_consensus.cc:3037] T d238a84052ad401cb969133851c14120 P 16fcd7a8ee8a47d2926ab4f3db9b48f6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:37:28.720054 15757 raft_consensus.cc:2454] T d238a84052ad401cb969133851c14120 P 601f7285b75f48539695ba51f9db205b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5f5cca3d04940e781e1bb7af2217ae1 in term 1.
I20260929 18:37:28.720181 15896 raft_consensus.cc:2454] T d238a84052ad401cb969133851c14120 P 16fcd7a8ee8a47d2926ab4f3db9b48f6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5f5cca3d04940e781e1bb7af2217ae1 in term 1.
I20260929 18:37:28.720381 15549 leader_election.cc:302] T d238a84052ad401cb969133851c14120 P f5f5cca3d04940e781e1bb7af2217ae1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 601f7285b75f48539695ba51f9db205b, f5f5cca3d04940e781e1bb7af2217ae1; no voters:
I20260929 18:37:28.720530 16030 raft_consensus.cc:2781] T d238a84052ad401cb969133851c14120 P f5f5cca3d04940e781e1bb7af2217ae1 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:37:28.720615 16030 raft_consensus.cc:686] T d238a84052ad401cb969133851c14120 P f5f5cca3d04940e781e1bb7af2217ae1 [term 1 LEADER]: Becoming Leader. State: Replica: f5f5cca3d04940e781e1bb7af2217ae1, State: Running, Role: LEADER
I20260929 18:37:28.720772 16030 consensus_queue.cc:237] T d238a84052ad401cb969133851c14120 P f5f5cca3d04940e781e1bb7af2217ae1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5f5cca3d04940e781e1bb7af2217ae1" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 46433 } } peers { permanent_uuid: "601f7285b75f48539695ba51f9db205b" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 35501 } } peers { permanent_uuid: "16fcd7a8ee8a47d2926ab4f3db9b48f6" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 35143 } }
I20260929 18:37:28.722437 15460 catalog_manager.cc:5495] T d238a84052ad401cb969133851c14120 P f5f5cca3d04940e781e1bb7af2217ae1 reported cstate change: term changed from 0 to 1, leader changed from <none> to f5f5cca3d04940e781e1bb7af2217ae1 (127.13.100.1). New cstate: current_term: 1 leader_uuid: "f5f5cca3d04940e781e1bb7af2217ae1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5f5cca3d04940e781e1bb7af2217ae1" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 46433 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "601f7285b75f48539695ba51f9db205b" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 35501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16fcd7a8ee8a47d2926ab4f3db9b48f6" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 35143 } health_report { overall_health: UNKNOWN } } }
I20260929 18:37:28.800396 16080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:28.797158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58426 (local address 127.13.100.1:46433)
0929 18:37:28.797401 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:28.797407 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:28.797422 (+ 15us) server_negotiation.cc:407] Connection header received
0929 18:37:28.797598 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:28.797600 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:28.797643 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:28.797727 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:37:28.797998 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:28.798505 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:28.799330 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:28.799552 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:28.799655 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:28.799882 (+ 227us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:37:28.799981 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:37:28.800101 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:28.800223 (+ 122us) server_negotiation.cc:299] Negotiation successful
0929 18:37:28.800270 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":120,"threads_started":1}
I20260929 18:37:28.806243 15896 raft_consensus.cc:1261] T d238a84052ad401cb969133851c14120 P 16fcd7a8ee8a47d2926ab4f3db9b48f6 [term 1 FOLLOWER]: Refusing update from remote peer f5f5cca3d04940e781e1bb7af2217ae1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:37:28.806372 15756 raft_consensus.cc:1261] T d238a84052ad401cb969133851c14120 P 601f7285b75f48539695ba51f9db205b [term 1 FOLLOWER]: Refusing update from remote peer f5f5cca3d04940e781e1bb7af2217ae1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:37:28.806695 16030 consensus_queue.cc:1035] T d238a84052ad401cb969133851c14120 P f5f5cca3d04940e781e1bb7af2217ae1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16fcd7a8ee8a47d2926ab4f3db9b48f6" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 35143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:37:28.806960 16035 consensus_queue.cc:1035] T d238a84052ad401cb969133851c14120 P f5f5cca3d04940e781e1bb7af2217ae1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "601f7285b75f48539695ba51f9db205b" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 35501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 18:37:28.820178 16028 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260929 18:37:30.388473 15460 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39224:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:37:30.388623 15460 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39224:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:37:30.391103 15460 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bb338bdc225b47fca102578584452513: Sending DeleteTablet for 3 replicas of tablet d238a84052ad401cb969133851c14120
I20260929 18:37:30.391601 15597 tablet_service.cc:1514] Processing DeleteTablet for tablet d238a84052ad401cb969133851c14120 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:37:30 UTC) from {username='kudu', principal='kudu/127.13.100.62@KRBTEST.COM'} at 127.0.0.1:58408
I20260929 18:37:30.391780 15737 tablet_service.cc:1514] Processing DeleteTablet for tablet d238a84052ad401cb969133851c14120 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:37:30 UTC) from {username='kudu', principal='kudu/127.13.100.62@KRBTEST.COM'} at 127.0.0.1:41276
I20260929 18:37:30.391880 15877 tablet_service.cc:1514] Processing DeleteTablet for tablet d238a84052ad401cb969133851c14120 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:37:30 UTC) from {username='kudu', principal='kudu/127.13.100.62@KRBTEST.COM'} at 127.0.0.1:39880
I20260929 18:37:30.391948 16105 tablet_replica.cc:330] T d238a84052ad401cb969133851c14120 P f5f5cca3d04940e781e1bb7af2217ae1: stopping tablet replica
I20260929 18:37:30.392073 16105 raft_consensus.cc:2229] T d238a84052ad401cb969133851c14120 P f5f5cca3d04940e781e1bb7af2217ae1 [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:37:30.392107 16106 tablet_replica.cc:330] T d238a84052ad401cb969133851c14120 P 601f7285b75f48539695ba51f9db205b: stopping tablet replica
I20260929 18:37:30.392220 16106 raft_consensus.cc:2229] T d238a84052ad401cb969133851c14120 P 601f7285b75f48539695ba51f9db205b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:37:30.392213 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 15532
I20260929 18:37:30.393263 16106 raft_consensus.cc:2258] T d238a84052ad401cb969133851c14120 P 601f7285b75f48539695ba51f9db205b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:37:30.393610 16107 tablet_replica.cc:330] T d238a84052ad401cb969133851c14120 P 16fcd7a8ee8a47d2926ab4f3db9b48f6: stopping tablet replica
I20260929 18:37:30.393751 16107 raft_consensus.cc:2229] T d238a84052ad401cb969133851c14120 P 16fcd7a8ee8a47d2926ab4f3db9b48f6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:37:30.393867 16107 raft_consensus.cc:2258] T d238a84052ad401cb969133851c14120 P 16fcd7a8ee8a47d2926ab4f3db9b48f6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:37:30.394081 16106 ts_tablet_manager.cc:1905] T d238a84052ad401cb969133851c14120 P 601f7285b75f48539695ba51f9db205b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:37:30.394742 16107 ts_tablet_manager.cc:1905] T d238a84052ad401cb969133851c14120 P 16fcd7a8ee8a47d2926ab4f3db9b48f6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:37:30.396864 16106 ts_tablet_manager.cc:1918] T d238a84052ad401cb969133851c14120 P 601f7285b75f48539695ba51f9db205b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 18:37:30.396963 16106 log.cc:1196] T d238a84052ad401cb969133851c14120 P 601f7285b75f48539695ba51f9db205b: Deleting WAL directory at /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707036041078-13712-0/minicluster-data/ts-1/wal/wals/d238a84052ad401cb969133851c14120
I20260929 18:37:30.397250 16106 ts_tablet_manager.cc:1939] T d238a84052ad401cb969133851c14120 P 601f7285b75f48539695ba51f9db205b: Deleting consensus metadata
I20260929 18:37:30.397406 16107 ts_tablet_manager.cc:1918] T d238a84052ad401cb969133851c14120 P 16fcd7a8ee8a47d2926ab4f3db9b48f6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 18:37:30.397475 16107 log.cc:1196] T d238a84052ad401cb969133851c14120 P 16fcd7a8ee8a47d2926ab4f3db9b48f6: Deleting WAL directory at /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707036041078-13712-0/minicluster-data/ts-2/wal/wals/d238a84052ad401cb969133851c14120
I20260929 18:37:30.397738 16107 ts_tablet_manager.cc:1939] T d238a84052ad401cb969133851c14120 P 16fcd7a8ee8a47d2926ab4f3db9b48f6: Deleting consensus metadata
I20260929 18:37:30.398030 15445 catalog_manager.cc:4841] TS 601f7285b75f48539695ba51f9db205b (127.13.100.2:35501): tablet d238a84052ad401cb969133851c14120 (table test-table [id=996f545c5c194ded81171c11a3f0a8ad]) successfully deleted
I20260929 18:37:30.398439 15445 catalog_manager.cc:4841] TS 16fcd7a8ee8a47d2926ab4f3db9b48f6 (127.13.100.3:35143): tablet d238a84052ad401cb969133851c14120 (table test-table [id=996f545c5c194ded81171c11a3f0a8ad]) successfully deleted
W20260929 18:37:30.400190 15444 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.100.1:46433 (error 108)
I20260929 18:37:30.400462 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 15672
I20260929 18:37:30.402421 16109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:30.401805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.1:46433 (local address 127.0.0.1:58434)
0929 18:37:30.402206 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:30.402292 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.1:46433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":130,"threads_started":1}
W20260929 18:37:30.402575 15444 catalog_manager.cc:4584] TS f5f5cca3d04940e781e1bb7af2217ae1 (127.13.100.1:46433): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d238a84052ad401cb969133851c14120: Network error: Client connection negotiation failed: client connection to 127.13.100.1:46433: connect: Connection refused (error 111)
I20260929 18:37:30.407415 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 15812
I20260929 18:37:30.414270 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 15428
2026-09-29T18:37:30Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (7002 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707036041078-13712-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:37:30 dist-test-slave-j1xs krb5kdc[16112](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:37:30 dist-test-slave-j1xs krb5kdc[16112](info): set up 2 sockets
Sep 29 18:37:30 dist-test-slave-j1xs krb5kdc[16112](info): commencing operation
krb5kdc: starting...
W20260929 18:37:31.453910 13712 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707036041078-13712-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707036041078-13712-0/krb5kdc/test-user.keytab.
Sep 29 18:37:31 dist-test-slave-j1xs krb5kdc[16112](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707051, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:37:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:37:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.100.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.100.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.100.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.62.keytab.
Entry for principal kudu/127.13.100.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.62.keytab.
I20260929 18:37:31.571204 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707036041078-13712-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707036041078-13712-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707036041078-13712-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.100.62:45775
--webserver_interface=127.13.100.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.62.keytab
--principal=kudu/127.13.100.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.100.20:42055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.100.62:45775
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707036041078-13712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707036041078-13712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707036041078-13712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:37:31.685032 16125 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:37:31.685348 16125 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:37:31.685427 16125 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:37:31.689664 16125 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:37:31.689769 16125 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:37:31.689805 16125 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:37:31.689863 16125 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:37:31.689917 16125 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:37:31.695247 16125 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.100.20:42055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707036041078-13712-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707036041078-13712-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.100.62:45775
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.62.keytab
--principal=kudu/127.13.100.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.100.62:45775
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.100.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16125
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707036041078-13712-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:37:31.696456 16125 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:37:31.697460 16125 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:37:31.706700 16133 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:37:31.706802 16125 server_base.cc:1034] running on GCE node
W20260929 18:37:31.706705 16131 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:37:31.706687 16130 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:37:31.707573 16125 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:37:31.708657 16125 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:37:31.709868 16125 hybrid_clock.cc:648] HybridClock initialized: now 1790707051709830 us; error 43 us; skew 500 ppm
Sep 29 18:37:31 dist-test-slave-j1xs krb5kdc[16112](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707051, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:37:31.712677 16125 init.cc:377] Logged in from keytab as kudu/127.13.100.62@KRBTEST.COM (short username kudu)
I20260929 18:37:31.713558 16125 webserver.cc:460] Webserver started at http://127.13.100.62:37005/ using document root <none> and password file <none>
I20260929 18:37:31.714162 16125 fs_manager.cc:359] Metadata directory not provided
I20260929 18:37:31.714226 16125 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:37:31.714396 16125 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:37:31.716362 16125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707036041078-13712-0/minicluster-data/master-0/data/instance:
uuid: "6f4f779c045d44108d298a2e1e756d61"
format_stamp: "Formatted at 2026-09-29 18:37:31 on dist-test-slave-j1xs"
server_key: "104000ece5f51d672a41b1158db21956"
server_key_iv: "ebc8b710aee57510e7566c444e15496c"
server_key_version: "encryptionkey@0"
I20260929 18:37:31.716926 16125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707036041078-13712-0/minicluster-data/master-0/wal/instance:
uuid: "6f4f779c045d44108d298a2e1e756d61"
format_stamp: "Formatted at 2026-09-29 18:37:31 on dist-test-slave-j1xs"
server_key: "104000ece5f51d672a41b1158db21956"
server_key_iv: "ebc8b710aee57510e7566c444e15496c"
server_key_version: "encryptionkey@0"
I20260929 18:37:31.720582 16125 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 18:37:31.722926 16140 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:37:31.724129 16125 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:37:31.724255 16125 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707036041078-13712-0/minicluster-data/master-0/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707036041078-13712-0/minicluster-data/master-0/wal
uuid: "6f4f779c045d44108d298a2e1e756d61"
format_stamp: "Formatted at 2026-09-29 18:37:31 on dist-test-slave-j1xs"
server_key: "104000ece5f51d672a41b1158db21956"
server_key_iv: "ebc8b710aee57510e7566c444e15496c"
server_key_version: "encryptionkey@0"
I20260929 18:37:31.724361 16125 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707036041078-13712-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707036041078-13712-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707036041078-13712-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:37:31.743738 16125 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:37:31.747159 16125 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:37:31.747356 16125 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:37:31.755291 16125 rpc_server.cc:307] RPC server started. Bound to: 127.13.100.62:45775
I20260929 18:37:31.755309 16192 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.100.62:45775 every 8 connection(s)
I20260929 18:37:31.756244 16125 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
I20260929 18:37:31.757336 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 16125
I20260929 18:37:31.757421 13712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707036041078-13712-0/minicluster-data/master-0/wal/instance
I20260929 18:37:31.757701 13712 external_mini_cluster.cc:1428] Setting key 3a6a2ac6cfdf374d006b9b3fa798337c
I20260929 18:37:31.759210 16193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:37:31.766103 16193 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f4f779c045d44108d298a2e1e756d61: Bootstrap starting.
Sep 29 18:37:31 dist-test-slave-j1xs krb5kdc[16112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707051, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.100.62@KRBTEST.COM
I20260929 18:37:31.768867 16193 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6f4f779c045d44108d298a2e1e756d61: Neither blocks nor log segments found. Creating new log.
I20260929 18:37:31.769819 16193 log.cc:824] T 00000000000000000000000000000000 P 6f4f779c045d44108d298a2e1e756d61: Log is configured to *not* fsync() on all Append() calls
I20260929 18:37:31.772224 16193 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f4f779c045d44108d298a2e1e756d61: No bootstrap required, opened a new log
I20260929 18:37:31.773748 16196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:31.759133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46854 (local address 127.13.100.62:45775)
0929 18:37:31.759581 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:31.759599 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:31.759640 (+ 41us) server_negotiation.cc:407] Connection header received
0929 18:37:31.760458 (+ 818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:31.760479 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:31.760887 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:31.761291 (+ 404us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:31.761987 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:31.763478 (+ 1491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:31.764333 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:31.765027 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:31.765198 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:31.767911 (+ 2713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:31.767917 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:31.767932 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:37:31.767964 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:31.770535 (+ 2571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:31.771053 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:31.771058 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:31.771064 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:31.771204 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:31.771484 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:31.771487 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:31.771489 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:31.771969 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:31.772171 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:31.772421 (+ 250us) server_negotiation.cc:299] Negotiation successful
0929 18:37:31.772830 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":152,"threads_started":1}
I20260929 18:37:31.776605 16193 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6f4f779c045d44108d298a2e1e756d61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f4f779c045d44108d298a2e1e756d61" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 45775 } }
I20260929 18:37:31.776830 16193 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6f4f779c045d44108d298a2e1e756d61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:37:31.776866 16193 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6f4f779c045d44108d298a2e1e756d61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f4f779c045d44108d298a2e1e756d61, State: Initialized, Role: FOLLOWER
I20260929 18:37:31.777208 16193 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6f4f779c045d44108d298a2e1e756d61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f4f779c045d44108d298a2e1e756d61" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 45775 } }
I20260929 18:37:31.777348 16193 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6f4f779c045d44108d298a2e1e756d61 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:37:31.777411 16193 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6f4f779c045d44108d298a2e1e756d61 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:37:31.777509 16193 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6f4f779c045d44108d298a2e1e756d61 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:37:31.778679 16193 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6f4f779c045d44108d298a2e1e756d61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f4f779c045d44108d298a2e1e756d61" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 45775 } }
I20260929 18:37:31.779026 16193 leader_election.cc:302] T 00000000000000000000000000000000 P 6f4f779c045d44108d298a2e1e756d61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6f4f779c045d44108d298a2e1e756d61; no voters:
I20260929 18:37:31.779331 16193 leader_election.cc:288] T 00000000000000000000000000000000 P 6f4f779c045d44108d298a2e1e756d61 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:37:31.779464 16198 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6f4f779c045d44108d298a2e1e756d61 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:37:31.779717 16198 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6f4f779c045d44108d298a2e1e756d61 [term 1 LEADER]: Becoming Leader. State: Replica: 6f4f779c045d44108d298a2e1e756d61, State: Running, Role: LEADER
I20260929 18:37:31.780061 16198 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6f4f779c045d44108d298a2e1e756d61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f4f779c045d44108d298a2e1e756d61" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 45775 } }
I20260929 18:37:31.780603 16193 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6f4f779c045d44108d298a2e1e756d61 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:37:31.781718 16199 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f4f779c045d44108d298a2e1e756d61 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6f4f779c045d44108d298a2e1e756d61" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f4f779c045d44108d298a2e1e756d61" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 45775 } } }
I20260929 18:37:31.781848 16199 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f4f779c045d44108d298a2e1e756d61 [sys.catalog]: This master's current role is: LEADER
I20260929 18:37:31.782253 16207 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:37:31.782256 16200 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f4f779c045d44108d298a2e1e756d61 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6f4f779c045d44108d298a2e1e756d61. Latest consensus state: current_term: 1 leader_uuid: "6f4f779c045d44108d298a2e1e756d61" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f4f779c045d44108d298a2e1e756d61" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 45775 } } }
I20260929 18:37:31.782925 16200 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f4f779c045d44108d298a2e1e756d61 [sys.catalog]: This master's current role is: LEADER
I20260929 18:37:31.785392 16207 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:37:31.791508 16207 catalog_manager.cc:1347] Generated new cluster ID: b33b563688174111a76845600bd793c0
I20260929 18:37:31.791639 16207 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:37:31.803653 16207 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:37:31.804908 16207 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:37:31.812317 16207 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6f4f779c045d44108d298a2e1e756d61: Generated new TSK 0
I20260929 18:37:31.813268 16207 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.100.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.100.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.100.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.1.keytab.
Entry for principal kudu/127.13.100.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.1.keytab.
I20260929 18:37:31.844975 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707036041078-13712-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707036041078-13712-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707036041078-13712-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707036041078-13712-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.100.1:0
--local_ip_for_outbound_sockets=127.13.100.1
--webserver_interface=127.13.100.1
--webserver_port=0
--tserver_master_addrs=127.13.100.62:45775
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.1.keytab
--principal=kudu/127.13.100.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.100.20:42055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707036041078-13712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707036041078-13712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707036041078-13712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:37:31.956468 16219 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:37:31.956720 16219 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:37:31.956791 16219 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:37:31.960877 16219 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:37:31.960932 16219 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:37:31.961014 16219 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.100.1
I20260929 18:37:31.966153 16219 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.100.20:42055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707036041078-13712-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707036041078-13712-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.1.keytab
--principal=kudu/127.13.100.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.100.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707036041078-13712-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.100.1
--webserver_port=0
--tserver_master_addrs=127.13.100.62:45775
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16219
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.100.1
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707036041078-13712-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:37:31.967172 16219 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:37:31.968202 16219 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:37:31.977478 16225 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:37:31.977486 16224 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:37:31.977555 16227 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:37:31.977715 16219 server_base.cc:1034] running on GCE node
I20260929 18:37:31.978178 16219 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:37:31.978804 16219 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:37:31.980041 16219 hybrid_clock.cc:648] HybridClock initialized: now 1790707051980008 us; error 48 us; skew 500 ppm
Sep 29 18:37:31 dist-test-slave-j1xs krb5kdc[16112](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707051, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:37:31.983026 16219 init.cc:377] Logged in from keytab as kudu/127.13.100.1@KRBTEST.COM (short username kudu)
I20260929 18:37:31.983781 16219 webserver.cc:460] Webserver started at http://127.13.100.1:43063/ using document root <none> and password file <none>
I20260929 18:37:31.984275 16219 fs_manager.cc:359] Metadata directory not provided
I20260929 18:37:31.984331 16219 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:37:31.984488 16219 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:37:31.986426 16219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707036041078-13712-0/minicluster-data/ts-0/data/instance:
uuid: "3efd49a05d804f97917ffb028b696c4a"
format_stamp: "Formatted at 2026-09-29 18:37:31 on dist-test-slave-j1xs"
server_key: "f642b3e2d5e4e6ae3bd70398da31493b"
server_key_iv: "537567f24d9acff50ba00d4b2d8627e7"
server_key_version: "encryptionkey@0"
I20260929 18:37:31.986923 16219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707036041078-13712-0/minicluster-data/ts-0/wal/instance:
uuid: "3efd49a05d804f97917ffb028b696c4a"
format_stamp: "Formatted at 2026-09-29 18:37:31 on dist-test-slave-j1xs"
server_key: "f642b3e2d5e4e6ae3bd70398da31493b"
server_key_iv: "537567f24d9acff50ba00d4b2d8627e7"
server_key_version: "encryptionkey@0"
I20260929 18:37:31.990422 16219 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 18:37:31.992695 16234 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:37:31.993863 16219 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:37:31.994004 16219 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707036041078-13712-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707036041078-13712-0/minicluster-data/ts-0/wal
uuid: "3efd49a05d804f97917ffb028b696c4a"
format_stamp: "Formatted at 2026-09-29 18:37:31 on dist-test-slave-j1xs"
server_key: "f642b3e2d5e4e6ae3bd70398da31493b"
server_key_iv: "537567f24d9acff50ba00d4b2d8627e7"
server_key_version: "encryptionkey@0"
I20260929 18:37:31.994123 16219 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707036041078-13712-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707036041078-13712-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707036041078-13712-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:37:32.032369 16219 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:37:32.035434 16219 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:37:32.035653 16219 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:37:32.036252 16219 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:37:32.037060 16219 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:37:32.037104 16219 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:37:32.037142 16219 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:37:32.037194 16219 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:37:32.046037 16219 rpc_server.cc:307] RPC server started. Bound to: 127.13.100.1:39683
I20260929 18:37:32.046068 16347 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.100.1:39683 every 8 connection(s)
I20260929 18:37:32.046860 16219 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707036041078-13712-0/minicluster-data/ts-0/data/info.pb
I20260929 18:37:32.051649 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 16219
I20260929 18:37:32.051779 13712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707036041078-13712-0/minicluster-data/ts-0/wal/instance
I20260929 18:37:32.052079 13712 external_mini_cluster.cc:1428] Setting key dc6899c8ffcecc8411fd29b2f01b6311
Sep 29 18:37:32 dist-test-slave-j1xs krb5kdc[16112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707051, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.1@KRBTEST.COM for kudu/127.13.100.62@KRBTEST.COM
I20260929 18:37:32.060833 16196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:32.048735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.1:43619 (local address 127.13.100.62:45775)
0929 18:37:32.048874 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:32.048878 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:32.049834 (+ 956us) server_negotiation.cc:407] Connection header received
0929 18:37:32.050709 (+ 875us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:32.050712 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:32.050755 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:32.050829 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:32.051990 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:32.052566 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:32.053658 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:32.053928 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:32.054038 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:32.057000 (+ 2962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:32.057003 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:32.057005 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:37:32.057026 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:32.058554 (+ 1528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:32.059196 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:32.059199 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:32.059200 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:32.059282 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:32.059710 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:32.059712 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:32.059714 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:32.059895 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:32.059976 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:32.060594 (+ 618us) server_negotiation.cc:299] Negotiation successful
0929 18:37:32.060713 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
WARNING: no policy specified for kudu/127.13.100.2@KRBTEST.COM; defaulting to no policy
I20260929 18:37:32.062022 16350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:32.049076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:45775 (local address 127.13.100.1:43619)
0929 18:37:32.049696 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:32.049734 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:32.050523 (+ 789us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:32.050974 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:32.050980 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:32.051301 (+ 321us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:37:32.051842 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:32.051861 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:32.052694 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:32.052698 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:32.053537 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:32.053545 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:32.054072 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:32.054076 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:32.054250 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:32.054892 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:37:32.054916 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:37:32.056843 (+ 1927us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:37:32.058682 (+ 1839us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:32.058690 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:32.058707 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:32.059083 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:32.059378 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:32.059382 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:32.059386 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:32.059585 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:32.060013 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:37:32.060021 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:37:32.060344 (+ 323us) client_negotiation.cc:769] Sending connection context
0929 18:37:32.060574 (+ 230us) client_negotiation.cc:240] Negotiation successful
0929 18:37:32.060916 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":107,"threads_started":1}
I20260929 18:37:32.063189 16348 heartbeater.cc:347] Connected to a master server at 127.13.100.62:45775
I20260929 18:37:32.063470 16348 heartbeater.cc:464] Registering TS with master...
I20260929 18:37:32.063992 16348 heartbeater.cc:511] Master 127.13.100.62:45775 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.100.2@KRBTEST.COM" created.
I20260929 18:37:32.065493 16157 ts_manager.cc:194] Registered new tserver with Master: 3efd49a05d804f97917ffb028b696c4a (127.13.100.1:39683)
I20260929 18:37:32.066715 16157 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.100.1@KRBTEST.COM'} at 127.13.100.1:43619
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.100.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.2.keytab.
Entry for principal kudu/127.13.100.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.2.keytab.
I20260929 18:37:32.080778 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707036041078-13712-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707036041078-13712-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707036041078-13712-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707036041078-13712-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.100.2:0
--local_ip_for_outbound_sockets=127.13.100.2
--webserver_interface=127.13.100.2
--webserver_port=0
--tserver_master_addrs=127.13.100.62:45775
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.2.keytab
--principal=kudu/127.13.100.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.100.20:42055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707036041078-13712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707036041078-13712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707036041078-13712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:37:32.187484 16353 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:37:32.187757 16353 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:37:32.187826 16353 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:37:32.191911 16353 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:37:32.191987 16353 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:37:32.192073 16353 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.100.2
I20260929 18:37:32.197294 16353 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.100.20:42055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707036041078-13712-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707036041078-13712-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.2.keytab
--principal=kudu/127.13.100.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.100.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707036041078-13712-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.100.2
--webserver_port=0
--tserver_master_addrs=127.13.100.62:45775
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16353
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.100.2
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707036041078-13712-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:37:32.198410 16353 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:37:32.199389 16353 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:37:32.208698 16359 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:37:32.208679 16361 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:37:32.208683 16358 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:37:32.208820 16353 server_base.cc:1034] running on GCE node
I20260929 18:37:32.209415 16353 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:37:32.210063 16353 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:37:32.211239 16353 hybrid_clock.cc:648] HybridClock initialized: now 1790707052211226 us; error 26 us; skew 500 ppm
Sep 29 18:37:32 dist-test-slave-j1xs krb5kdc[16112](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707052, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:37:32.214011 16353 init.cc:377] Logged in from keytab as kudu/127.13.100.2@KRBTEST.COM (short username kudu)
I20260929 18:37:32.214782 16353 webserver.cc:460] Webserver started at http://127.13.100.2:44017/ using document root <none> and password file <none>
I20260929 18:37:32.215262 16353 fs_manager.cc:359] Metadata directory not provided
I20260929 18:37:32.215317 16353 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:37:32.215466 16353 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:37:32.217310 16353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707036041078-13712-0/minicluster-data/ts-1/data/instance:
uuid: "997b0f23c74a4678a42707fa806f5fcd"
format_stamp: "Formatted at 2026-09-29 18:37:32 on dist-test-slave-j1xs"
server_key: "a68e6c4225c0002fe659fc91d8829163"
server_key_iv: "0aecc6cf35191659a3d74be027c36ef9"
server_key_version: "encryptionkey@0"
I20260929 18:37:32.217790 16353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707036041078-13712-0/minicluster-data/ts-1/wal/instance:
uuid: "997b0f23c74a4678a42707fa806f5fcd"
format_stamp: "Formatted at 2026-09-29 18:37:32 on dist-test-slave-j1xs"
server_key: "a68e6c4225c0002fe659fc91d8829163"
server_key_iv: "0aecc6cf35191659a3d74be027c36ef9"
server_key_version: "encryptionkey@0"
I20260929 18:37:32.221235 16353 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 18:37:32.223618 16368 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:37:32.224788 16353 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 18:37:32.224933 16353 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707036041078-13712-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707036041078-13712-0/minicluster-data/ts-1/wal
uuid: "997b0f23c74a4678a42707fa806f5fcd"
format_stamp: "Formatted at 2026-09-29 18:37:32 on dist-test-slave-j1xs"
server_key: "a68e6c4225c0002fe659fc91d8829163"
server_key_iv: "0aecc6cf35191659a3d74be027c36ef9"
server_key_version: "encryptionkey@0"
I20260929 18:37:32.225044 16353 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707036041078-13712-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707036041078-13712-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707036041078-13712-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:37:32.266635 16353 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:37:32.269955 16353 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:37:32.270169 16353 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:37:32.270756 16353 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:37:32.271608 16353 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:37:32.271677 16353 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:37:32.271746 16353 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:37:32.271795 16353 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:37:32.281211 16481 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.100.2:42293 every 8 connection(s)
I20260929 18:37:32.281217 16353 rpc_server.cc:307] RPC server started. Bound to: 127.13.100.2:42293
I20260929 18:37:32.282403 16353 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707036041078-13712-0/minicluster-data/ts-1/data/info.pb
I20260929 18:37:32.287212 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 16353
I20260929 18:37:32.287323 13712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707036041078-13712-0/minicluster-data/ts-1/wal/instance
I20260929 18:37:32.287588 13712 external_mini_cluster.cc:1428] Setting key 8ca446680fea2a05cc73d6bbf2a8bb49
Sep 29 18:37:32 dist-test-slave-j1xs krb5kdc[16112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707052, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.2@KRBTEST.COM for kudu/127.13.100.62@KRBTEST.COM
I20260929 18:37:32.297003 16196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:32.284289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.2:47347 (local address 127.13.100.62:45775)
0929 18:37:32.284460 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:32.284465 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:32.285507 (+ 1042us) server_negotiation.cc:407] Connection header received
0929 18:37:32.286413 (+ 906us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:32.286416 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:32.286461 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:32.286534 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:32.288034 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:32.288777 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:32.289953 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:32.290196 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:32.290306 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:32.292999 (+ 2693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:32.293006 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:32.293013 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:37:32.293047 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:32.294814 (+ 1767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:32.295346 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:32.295351 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:32.295354 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:32.295449 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:32.295796 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:32.295799 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:32.295801 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:32.296038 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:32.296153 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:32.296615 (+ 462us) server_negotiation.cc:299] Negotiation successful
0929 18:37:32.296736 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
WARNING: no policy specified for kudu/127.13.100.3@KRBTEST.COM; defaulting to no policy
I20260929 18:37:32.297857 16484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:32.284715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:45775 (local address 127.13.100.2:47347)
0929 18:37:32.285356 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:32.285395 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:32.286226 (+ 831us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:32.286700 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:32.286707 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:32.287172 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:37:32.287851 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:32.287870 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:32.288910 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:32.288914 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:32.289786 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:32.289795 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:32.290340 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:32.290344 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:32.290549 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:32.291232 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:37:32.291248 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:37:32.292794 (+ 1546us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:37:32.294956 (+ 2162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:32.294961 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:32.294972 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:32.295242 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:32.295542 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:32.295544 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:32.295545 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:32.295702 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:32.296144 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:37:32.296148 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:37:32.296377 (+ 229us) client_negotiation.cc:769] Sending connection context
0929 18:37:32.296532 (+ 155us) client_negotiation.cc:240] Negotiation successful
0929 18:37:32.296914 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":113,"threads_started":1}
I20260929 18:37:32.299075 16482 heartbeater.cc:347] Connected to a master server at 127.13.100.62:45775
I20260929 18:37:32.299319 16482 heartbeater.cc:464] Registering TS with master...
I20260929 18:37:32.299785 16482 heartbeater.cc:511] Master 127.13.100.62:45775 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.100.3@KRBTEST.COM" created.
I20260929 18:37:32.300822 16157 ts_manager.cc:194] Registered new tserver with Master: 997b0f23c74a4678a42707fa806f5fcd (127.13.100.2:42293)
I20260929 18:37:32.301381 16157 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.100.2@KRBTEST.COM'} at 127.13.100.2:47347
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.100.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.3.keytab.
Entry for principal kudu/127.13.100.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.3.keytab.
I20260929 18:37:32.315158 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707036041078-13712-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707036041078-13712-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707036041078-13712-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707036041078-13712-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.100.3:0
--local_ip_for_outbound_sockets=127.13.100.3
--webserver_interface=127.13.100.3
--webserver_port=0
--tserver_master_addrs=127.13.100.62:45775
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.3.keytab
--principal=kudu/127.13.100.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.100.20:42055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707036041078-13712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707036041078-13712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707036041078-13712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:37:32.424983 16487 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:37:32.425241 16487 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:37:32.425313 16487 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:37:32.429847 16487 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:37:32.429960 16487 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:37:32.430064 16487 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.100.3
I20260929 18:37:32.435425 16487 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.100.20:42055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707036041078-13712-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707036041078-13712-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.3.keytab
--principal=kudu/127.13.100.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.100.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707036041078-13712-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.100.3
--webserver_port=0
--tserver_master_addrs=127.13.100.62:45775
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16487
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.100.3
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707036041078-13712-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:37:32.436419 16487 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:37:32.437445 16487 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:37:32.447132 16492 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:37:32.447211 16495 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:37:32.447257 16487 server_base.cc:1034] running on GCE node
W20260929 18:37:32.447207 16493 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:37:32.447896 16487 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:37:32.448534 16487 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:37:32.449724 16487 hybrid_clock.cc:648] HybridClock initialized: now 1790707052449711 us; error 33 us; skew 500 ppm
Sep 29 18:37:32 dist-test-slave-j1xs krb5kdc[16112](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707052, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:37:32.452544 16487 init.cc:377] Logged in from keytab as kudu/127.13.100.3@KRBTEST.COM (short username kudu)
I20260929 18:37:32.453372 16487 webserver.cc:460] Webserver started at http://127.13.100.3:42519/ using document root <none> and password file <none>
I20260929 18:37:32.453871 16487 fs_manager.cc:359] Metadata directory not provided
I20260929 18:37:32.453989 16487 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:37:32.454156 16487 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:37:32.456061 16487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707036041078-13712-0/minicluster-data/ts-2/data/instance:
uuid: "69c284bbe9224757b068342308880b3e"
format_stamp: "Formatted at 2026-09-29 18:37:32 on dist-test-slave-j1xs"
server_key: "477b3489e551b3e543c8b5fcbd0931e7"
server_key_iv: "8c07283459c3abe5d071cf0f10ecfbce"
server_key_version: "encryptionkey@0"
I20260929 18:37:32.456571 16487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707036041078-13712-0/minicluster-data/ts-2/wal/instance:
uuid: "69c284bbe9224757b068342308880b3e"
format_stamp: "Formatted at 2026-09-29 18:37:32 on dist-test-slave-j1xs"
server_key: "477b3489e551b3e543c8b5fcbd0931e7"
server_key_iv: "8c07283459c3abe5d071cf0f10ecfbce"
server_key_version: "encryptionkey@0"
I20260929 18:37:32.460103 16487 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260929 18:37:32.462370 16502 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:37:32.463526 16487 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:37:32.463666 16487 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707036041078-13712-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707036041078-13712-0/minicluster-data/ts-2/wal
uuid: "69c284bbe9224757b068342308880b3e"
format_stamp: "Formatted at 2026-09-29 18:37:32 on dist-test-slave-j1xs"
server_key: "477b3489e551b3e543c8b5fcbd0931e7"
server_key_iv: "8c07283459c3abe5d071cf0f10ecfbce"
server_key_version: "encryptionkey@0"
I20260929 18:37:32.463760 16487 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707036041078-13712-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707036041078-13712-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707036041078-13712-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:37:32.485074 16487 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:37:32.488265 16487 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:37:32.488477 16487 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:37:32.489068 16487 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:37:32.489842 16487 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:37:32.489905 16487 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:37:32.490010 16487 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:37:32.490049 16487 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:37:32.498857 16487 rpc_server.cc:307] RPC server started. Bound to: 127.13.100.3:46737
I20260929 18:37:32.498893 16615 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.100.3:46737 every 8 connection(s)
I20260929 18:37:32.499673 16487 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707036041078-13712-0/minicluster-data/ts-2/data/info.pb
I20260929 18:37:32.501497 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 16487
I20260929 18:37:32.501626 13712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707036041078-13712-0/minicluster-data/ts-2/wal/instance
I20260929 18:37:32.501977 13712 external_mini_cluster.cc:1428] Setting key 6d511ea3cf7b99cf69e29fd697231bcd
Sep 29 18:37:32 dist-test-slave-j1xs krb5kdc[16112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707052, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.3@KRBTEST.COM for kudu/127.13.100.62@KRBTEST.COM
I20260929 18:37:32.512605 16196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:32.501291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.3:57123 (local address 127.13.100.62:45775)
0929 18:37:32.501444 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:32.501450 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:32.502622 (+ 1172us) server_negotiation.cc:407] Connection header received
0929 18:37:32.503699 (+ 1077us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:32.503702 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:32.503753 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:32.503826 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:32.504937 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:32.505464 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:32.506771 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:32.506999 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:32.507106 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:32.509443 (+ 2337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:32.509447 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:32.509455 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:37:32.509476 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:32.510726 (+ 1250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:32.511212 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:32.511215 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:32.511216 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:32.511283 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:32.511604 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:32.511607 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:32.511608 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:32.511773 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:32.511876 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:32.512327 (+ 451us) server_negotiation.cc:299] Negotiation successful
0929 18:37:32.512439 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 18:37:32.513345 16618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:32.501715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:45775 (local address 127.13.100.3:57123)
0929 18:37:32.502447 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:32.502513 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:32.503510 (+ 997us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:32.503981 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:32.503991 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:32.504351 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:37:32.504803 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:32.504816 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:32.505582 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:32.505586 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:32.506645 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:32.506654 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:32.507136 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:32.507140 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:32.507298 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:32.507849 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:37:32.507864 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:37:32.509293 (+ 1429us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:37:32.510834 (+ 1541us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:32.510840 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:32.510853 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:32.511090 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:32.511370 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:32.511374 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:32.511376 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:32.511503 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:32.511872 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:37:32.511878 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:37:32.512102 (+ 224us) client_negotiation.cc:769] Sending connection context
0929 18:37:32.512286 (+ 184us) client_negotiation.cc:240] Negotiation successful
0929 18:37:32.512575 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":147,"threads_started":1}
I20260929 18:37:32.514590 16616 heartbeater.cc:347] Connected to a master server at 127.13.100.62:45775
I20260929 18:37:32.514823 16616 heartbeater.cc:464] Registering TS with master...
I20260929 18:37:32.515286 16616 heartbeater.cc:511] Master 127.13.100.62:45775 requested a full tablet report, sending...
I20260929 18:37:32.516288 16157 ts_manager.cc:194] Registered new tserver with Master: 69c284bbe9224757b068342308880b3e (127.13.100.3:46737)
I20260929 18:37:32.516811 16157 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.100.3@KRBTEST.COM'} at 127.13.100.3:57123
I20260929 18:37:32.527426 13712 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260929 18:37:32.539664 16196 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:37:32.538397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46858 (local address 127.13.100.62:45775)
0929 18:37:32.538518 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:32.538522 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:32.538709 (+ 187us) server_negotiation.cc:407] Connection header received
0929 18:37:32.538821 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:32.538824 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:32.538862 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:32.538939 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:32.539563 (+ 624us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46858: BlockingRecv error: recv got EOF from 127.0.0.1:46858 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 18:37:32.540755 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 16219
I20260929 18:37:32.545981 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 16353
I20260929 18:37:32.550772 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 16487
I20260929 18:37:32.555831 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 16125
2026-09-29T18:37:32Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2139 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790707036041078-13712-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:37:32 dist-test-slave-j1xs krb5kdc[16628](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:37:32 dist-test-slave-j1xs krb5kdc[16628](info): set up 2 sockets
Sep 29 18:37:32 dist-test-slave-j1xs krb5kdc[16628](info): commencing operation
krb5kdc: starting...
W20260929 18:37:33.594125 13712 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790707036041078-13712-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790707036041078-13712-0/krb5kdc/test-user.keytab.
Sep 29 18:37:33 dist-test-slave-j1xs krb5kdc[16628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707053, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:37:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:37:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.100.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.100.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.100.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.62.keytab.
Entry for principal kudu/127.13.100.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.62.keytab.
I20260929 18:37:33.714849 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790707036041078-13712-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790707036041078-13712-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790707036041078-13712-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.100.62:35497
--webserver_interface=127.13.100.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.62.keytab
--principal=kudu/127.13.100.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.100.20:39921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.100.62:35497
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790707036041078-13712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790707036041078-13712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790707036041078-13712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:37:33.822432 16641 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:37:33.822741 16641 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:37:33.822819 16641 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:37:33.826930 16641 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:37:33.827024 16641 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:37:33.827061 16641 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:37:33.827098 16641 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:37:33.827131 16641 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:37:33.832381 16641 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.100.20:39921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790707036041078-13712-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790707036041078-13712-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.100.62:35497
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.62.keytab
--principal=kudu/127.13.100.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.100.62:35497
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.100.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16641
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790707036041078-13712-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:37:33.833505 16641 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:37:33.834556 16641 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:37:33.843318 16646 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:37:33.843439 16641 server_base.cc:1034] running on GCE node
W20260929 18:37:33.843318 16649 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:37:33.843469 16647 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:37:33.844280 16641 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:37:33.845448 16641 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:37:33.846653 16641 hybrid_clock.cc:648] HybridClock initialized: now 1790707053846635 us; error 29 us; skew 500 ppm
Sep 29 18:37:33 dist-test-slave-j1xs krb5kdc[16628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707053, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:37:33.852774 16641 init.cc:377] Logged in from keytab as kudu/127.13.100.62@KRBTEST.COM (short username kudu)
I20260929 18:37:33.853694 16641 webserver.cc:460] Webserver started at http://127.13.100.62:34501/ using document root <none> and password file <none>
I20260929 18:37:33.854426 16641 fs_manager.cc:359] Metadata directory not provided
I20260929 18:37:33.854530 16641 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:37:33.854775 16641 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:37:33.856838 16641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790707036041078-13712-0/minicluster-data/master-0/data/instance:
uuid: "644c4384e71b462ab5bf89150a887493"
format_stamp: "Formatted at 2026-09-29 18:37:33 on dist-test-slave-j1xs"
server_key: "5760fe11f1216b7a6ce3480e67890a57"
server_key_iv: "ae2d79b90fca744d9e9e7d787d4a3749"
server_key_version: "encryptionkey@0"
I20260929 18:37:33.857417 16641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790707036041078-13712-0/minicluster-data/master-0/wal/instance:
uuid: "644c4384e71b462ab5bf89150a887493"
format_stamp: "Formatted at 2026-09-29 18:37:33 on dist-test-slave-j1xs"
server_key: "5760fe11f1216b7a6ce3480e67890a57"
server_key_iv: "ae2d79b90fca744d9e9e7d787d4a3749"
server_key_version: "encryptionkey@0"
I20260929 18:37:33.861191 16641 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260929 18:37:33.863770 16656 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:37:33.865082 16641 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260929 18:37:33.865242 16641 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790707036041078-13712-0/minicluster-data/master-0/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790707036041078-13712-0/minicluster-data/master-0/wal
uuid: "644c4384e71b462ab5bf89150a887493"
format_stamp: "Formatted at 2026-09-29 18:37:33 on dist-test-slave-j1xs"
server_key: "5760fe11f1216b7a6ce3480e67890a57"
server_key_iv: "ae2d79b90fca744d9e9e7d787d4a3749"
server_key_version: "encryptionkey@0"
I20260929 18:37:33.865361 16641 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790707036041078-13712-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790707036041078-13712-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790707036041078-13712-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:37:33.877821 16641 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:37:33.881119 16641 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:37:33.881337 16641 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:37:33.888662 16641 rpc_server.cc:307] RPC server started. Bound to: 127.13.100.62:35497
I20260929 18:37:33.888957 16708 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.100.62:35497 every 8 connection(s)
I20260929 18:37:33.889635 16641 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
I20260929 18:37:33.890671 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 16641
I20260929 18:37:33.890825 13712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790707036041078-13712-0/minicluster-data/master-0/wal/instance
I20260929 18:37:33.891114 13712 external_mini_cluster.cc:1428] Setting key 7d4ad43bdb0b415046c962244da3207d
I20260929 18:37:33.893116 16709 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:37:33.898744 16709 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 644c4384e71b462ab5bf89150a887493: Bootstrap starting.
Sep 29 18:37:33 dist-test-slave-j1xs krb5kdc[16628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707053, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.100.62@KRBTEST.COM
I20260929 18:37:33.900931 16709 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 644c4384e71b462ab5bf89150a887493: Neither blocks nor log segments found. Creating new log.
I20260929 18:37:33.901649 16709 log.cc:824] T 00000000000000000000000000000000 P 644c4384e71b462ab5bf89150a887493: Log is configured to *not* fsync() on all Append() calls
I20260929 18:37:33.903576 16709 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 644c4384e71b462ab5bf89150a887493: No bootstrap required, opened a new log
I20260929 18:37:33.905224 16712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:33.892553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39252 (local address 127.13.100.62:35497)
0929 18:37:33.893109 (+ 556us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:33.893129 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:33.893173 (+ 44us) server_negotiation.cc:407] Connection header received
0929 18:37:33.893844 (+ 671us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:33.893860 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:33.894264 (+ 404us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:33.894595 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:33.895457 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:33.896417 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:33.897279 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:33.897740 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:33.897925 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:33.900409 (+ 2484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:33.900413 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:33.900426 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:37:33.900452 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:33.902520 (+ 2068us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:33.902955 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:33.902960 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:33.902966 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:33.903081 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:33.903364 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:33.903368 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:33.903370 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:33.903798 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:33.904008 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:33.904254 (+ 246us) server_negotiation.cc:299] Negotiation successful
0929 18:37:33.904534 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":158,"threads_started":1}
I20260929 18:37:33.907539 16709 raft_consensus.cc:348] T 00000000000000000000000000000000 P 644c4384e71b462ab5bf89150a887493 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "644c4384e71b462ab5bf89150a887493" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 35497 } }
I20260929 18:37:33.907819 16709 raft_consensus.cc:374] T 00000000000000000000000000000000 P 644c4384e71b462ab5bf89150a887493 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:37:33.907886 16709 raft_consensus.cc:729] T 00000000000000000000000000000000 P 644c4384e71b462ab5bf89150a887493 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 644c4384e71b462ab5bf89150a887493, State: Initialized, Role: FOLLOWER
I20260929 18:37:33.908334 13712 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:37:33.908397 16709 consensus_queue.cc:260] T 00000000000000000000000000000000 P 644c4384e71b462ab5bf89150a887493 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "644c4384e71b462ab5bf89150a887493" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 35497 } }
I20260929 18:37:33.908583 16709 raft_consensus.cc:388] T 00000000000000000000000000000000 P 644c4384e71b462ab5bf89150a887493 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:37:33.908664 16709 raft_consensus.cc:482] T 00000000000000000000000000000000 P 644c4384e71b462ab5bf89150a887493 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:37:33.908746 16709 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 644c4384e71b462ab5bf89150a887493 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:37:33.909675 16709 raft_consensus.cc:504] T 00000000000000000000000000000000 P 644c4384e71b462ab5bf89150a887493 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "644c4384e71b462ab5bf89150a887493" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 35497 } }
I20260929 18:37:33.910064 16709 leader_election.cc:302] T 00000000000000000000000000000000 P 644c4384e71b462ab5bf89150a887493 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 644c4384e71b462ab5bf89150a887493; no voters:
I20260929 18:37:33.910458 16709 leader_election.cc:288] T 00000000000000000000000000000000 P 644c4384e71b462ab5bf89150a887493 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:37:33.910571 16715 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 644c4384e71b462ab5bf89150a887493 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:37:33.910822 16715 raft_consensus.cc:686] T 00000000000000000000000000000000 P 644c4384e71b462ab5bf89150a887493 [term 1 LEADER]: Becoming Leader. State: Replica: 644c4384e71b462ab5bf89150a887493, State: Running, Role: LEADER
I20260929 18:37:33.911140 16715 consensus_queue.cc:237] T 00000000000000000000000000000000 P 644c4384e71b462ab5bf89150a887493 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "644c4384e71b462ab5bf89150a887493" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 35497 } }
I20260929 18:37:33.911854 16709 sys_catalog.cc:564] T 00000000000000000000000000000000 P 644c4384e71b462ab5bf89150a887493 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:37:33.912164 16715 sys_catalog.cc:455] T 00000000000000000000000000000000 P 644c4384e71b462ab5bf89150a887493 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 644c4384e71b462ab5bf89150a887493. Latest consensus state: current_term: 1 leader_uuid: "644c4384e71b462ab5bf89150a887493" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "644c4384e71b462ab5bf89150a887493" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 35497 } } }
I20260929 18:37:33.912283 16715 sys_catalog.cc:458] T 00000000000000000000000000000000 P 644c4384e71b462ab5bf89150a887493 [sys.catalog]: This master's current role is: LEADER
I20260929 18:37:33.912684 16719 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:37:33.912706 16717 sys_catalog.cc:455] T 00000000000000000000000000000000 P 644c4384e71b462ab5bf89150a887493 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "644c4384e71b462ab5bf89150a887493" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "644c4384e71b462ab5bf89150a887493" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 35497 } } }
I20260929 18:37:33.912875 16717 sys_catalog.cc:458] T 00000000000000000000000000000000 P 644c4384e71b462ab5bf89150a887493 [sys.catalog]: This master's current role is: LEADER
I20260929 18:37:33.915874 16719 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:37:33.921862 16719 catalog_manager.cc:1347] Generated new cluster ID: bc70e7c364084b2d9395ab52e2178a4f
I20260929 18:37:33.923002 16719 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:37:33.929943 16712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:33.923858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39256 (local address 127.13.100.62:35497)
0929 18:37:33.923963 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:33.923967 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:33.924296 (+ 329us) server_negotiation.cc:407] Connection header received
0929 18:37:33.924427 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:33.924430 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:33.924490 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:33.924592 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:33.925217 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:33.926178 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:33.927371 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:33.927774 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:33.927887 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:33.928823 (+ 936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:33.928827 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:33.928830 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:37:33.929462 (+ 632us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:33.929493 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:37:33.929512 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:33.929599 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:33.929721 (+ 122us) server_negotiation.cc:299] Negotiation successful
0929 18:37:33.929777 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260929 18:37:33.933710 16719 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:37:33.934751 16719 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:37:33.947731 16719 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 644c4384e71b462ab5bf89150a887493: Generated new TSK 0
I20260929 18:37:33.948433 16719 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:37:33.955665 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 16641
2026-09-29T18:37:33Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1398 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707036041078-13712-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:37:34 dist-test-slave-j1xs krb5kdc[16742](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:37:34 dist-test-slave-j1xs krb5kdc[16742](info): set up 2 sockets
Sep 29 18:37:34 dist-test-slave-j1xs krb5kdc[16742](info): commencing operation
krb5kdc: starting...
W20260929 18:37:34.994112 13712 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707036041078-13712-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707036041078-13712-0/krb5kdc/test-user.keytab.
Sep 29 18:37:35 dist-test-slave-j1xs krb5kdc[16742](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707055, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:37:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:37:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.100.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.100.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.100.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.62.keytab.
Entry for principal kudu/127.13.100.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.62.keytab.
I20260929 18:37:35.116389 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707036041078-13712-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707036041078-13712-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707036041078-13712-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.100.62:33159
--webserver_interface=127.13.100.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.62.keytab
--principal=kudu/127.13.100.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.100.20:42491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.100.62:33159
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707036041078-13712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707036041078-13712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707036041078-13712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:37:35.231287 16755 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:37:35.231604 16755 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:37:35.231693 16755 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:37:35.235796 16755 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:37:35.235893 16755 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:37:35.235940 16755 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:37:35.236006 16755 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:37:35.236049 16755 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:37:35.241338 16755 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.100.20:42491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707036041078-13712-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707036041078-13712-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.100.62:33159
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.62.keytab
--principal=kudu/127.13.100.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.100.62:33159
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.100.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16755
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707036041078-13712-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:37:35.242553 16755 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:37:35.243615 16755 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:37:35.252534 16761 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:37:35.252578 16763 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:37:35.252561 16760 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:37:35.252662 16755 server_base.cc:1034] running on GCE node
I20260929 18:37:35.253320 16755 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:37:35.254395 16755 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:37:35.255554 16755 hybrid_clock.cc:648] HybridClock initialized: now 1790707055255536 us; error 29 us; skew 500 ppm
Sep 29 18:37:35 dist-test-slave-j1xs krb5kdc[16742](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707055, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:37:35.258350 16755 init.cc:377] Logged in from keytab as kudu/127.13.100.62@KRBTEST.COM (short username kudu)
I20260929 18:37:35.259120 16755 webserver.cc:460] Webserver started at http://127.13.100.62:42935/ using document root <none> and password file <none>
I20260929 18:37:35.259614 16755 fs_manager.cc:359] Metadata directory not provided
I20260929 18:37:35.259670 16755 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:37:35.259833 16755 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:37:35.261721 16755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707036041078-13712-0/minicluster-data/master-0/data/instance:
uuid: "7e3f1d175f7240cd9cdfea965742d943"
format_stamp: "Formatted at 2026-09-29 18:37:35 on dist-test-slave-j1xs"
server_key: "3e389134cfb083a973990ced073886a4"
server_key_iv: "919cc22b260884cdfb6bfd268bab3156"
server_key_version: "encryptionkey@0"
I20260929 18:37:35.262346 16755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707036041078-13712-0/minicluster-data/master-0/wal/instance:
uuid: "7e3f1d175f7240cd9cdfea965742d943"
format_stamp: "Formatted at 2026-09-29 18:37:35 on dist-test-slave-j1xs"
server_key: "3e389134cfb083a973990ced073886a4"
server_key_iv: "919cc22b260884cdfb6bfd268bab3156"
server_key_version: "encryptionkey@0"
I20260929 18:37:35.265806 16755 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:37:35.268198 16770 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:37:35.269429 16755 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:37:35.269563 16755 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707036041078-13712-0/minicluster-data/master-0/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707036041078-13712-0/minicluster-data/master-0/wal
uuid: "7e3f1d175f7240cd9cdfea965742d943"
format_stamp: "Formatted at 2026-09-29 18:37:35 on dist-test-slave-j1xs"
server_key: "3e389134cfb083a973990ced073886a4"
server_key_iv: "919cc22b260884cdfb6bfd268bab3156"
server_key_version: "encryptionkey@0"
I20260929 18:37:35.269672 16755 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707036041078-13712-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707036041078-13712-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707036041078-13712-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:37:35.286718 16755 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:37:35.290012 16755 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:37:35.290207 16755 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:37:35.297036 16755 rpc_server.cc:307] RPC server started. Bound to: 127.13.100.62:33159
I20260929 18:37:35.297344 16822 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.100.62:33159 every 8 connection(s)
I20260929 18:37:35.298004 16755 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
I20260929 18:37:35.300688 16823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:37:35.303035 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 16755
I20260929 18:37:35.303169 13712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707036041078-13712-0/minicluster-data/master-0/wal/instance
I20260929 18:37:35.303431 13712 external_mini_cluster.cc:1428] Setting key 1412bb1ee59aa98359b326c72d12ac8e
I20260929 18:37:35.306581 16823 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e3f1d175f7240cd9cdfea965742d943: Bootstrap starting.
I20260929 18:37:35.308733 16823 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7e3f1d175f7240cd9cdfea965742d943: Neither blocks nor log segments found. Creating new log.
I20260929 18:37:35.309458 16823 log.cc:824] T 00000000000000000000000000000000 P 7e3f1d175f7240cd9cdfea965742d943: Log is configured to *not* fsync() on all Append() calls
I20260929 18:37:35.311252 16823 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e3f1d175f7240cd9cdfea965742d943: No bootstrap required, opened a new log
Sep 29 18:37:35 dist-test-slave-j1xs krb5kdc[16742](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707055, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.100.62@KRBTEST.COM
I20260929 18:37:35.314462 16823 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7e3f1d175f7240cd9cdfea965742d943 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e3f1d175f7240cd9cdfea965742d943" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 33159 } }
I20260929 18:37:35.314694 16823 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7e3f1d175f7240cd9cdfea965742d943 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:37:35.314777 16823 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7e3f1d175f7240cd9cdfea965742d943 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e3f1d175f7240cd9cdfea965742d943, State: Initialized, Role: FOLLOWER
I20260929 18:37:35.315196 16823 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7e3f1d175f7240cd9cdfea965742d943 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e3f1d175f7240cd9cdfea965742d943" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 33159 } }
I20260929 18:37:35.315341 16823 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7e3f1d175f7240cd9cdfea965742d943 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:37:35.315431 16823 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7e3f1d175f7240cd9cdfea965742d943 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:37:35.315544 16823 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7e3f1d175f7240cd9cdfea965742d943 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:37:35.316360 16823 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7e3f1d175f7240cd9cdfea965742d943 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e3f1d175f7240cd9cdfea965742d943" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 33159 } }
I20260929 18:37:35.316694 16823 leader_election.cc:302] T 00000000000000000000000000000000 P 7e3f1d175f7240cd9cdfea965742d943 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7e3f1d175f7240cd9cdfea965742d943; no voters:
I20260929 18:37:35.317013 16823 leader_election.cc:288] T 00000000000000000000000000000000 P 7e3f1d175f7240cd9cdfea965742d943 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:37:35.317142 16828 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7e3f1d175f7240cd9cdfea965742d943 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:37:35.317333 16828 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7e3f1d175f7240cd9cdfea965742d943 [term 1 LEADER]: Becoming Leader. State: Replica: 7e3f1d175f7240cd9cdfea965742d943, State: Running, Role: LEADER
I20260929 18:37:35.317615 16828 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7e3f1d175f7240cd9cdfea965742d943 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e3f1d175f7240cd9cdfea965742d943" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 33159 } }
I20260929 18:37:35.318181 16826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:35.304793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54134 (local address 127.13.100.62:33159)
0929 18:37:35.305258 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:35.305271 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:35.305308 (+ 37us) server_negotiation.cc:407] Connection header received
0929 18:37:35.306026 (+ 718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:35.306052 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:35.306385 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:35.306723 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:35.307542 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:35.308507 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:35.309398 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:35.309966 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:35.310131 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:35.312640 (+ 2509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:35.312646 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:35.312667 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:37:35.312692 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:35.315121 (+ 2429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:35.315612 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:35.315617 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:35.315623 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:35.315754 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:35.316049 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:35.316052 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:35.316054 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:35.316426 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:35.316620 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:35.316901 (+ 281us) server_negotiation.cc:299] Negotiation successful
0929 18:37:35.317206 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":153,"threads_started":1}
I20260929 18:37:35.318272 16823 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7e3f1d175f7240cd9cdfea965742d943 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:37:35.320130 16829 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e3f1d175f7240cd9cdfea965742d943 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7e3f1d175f7240cd9cdfea965742d943" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e3f1d175f7240cd9cdfea965742d943" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 33159 } } }
I20260929 18:37:35.320243 16829 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e3f1d175f7240cd9cdfea965742d943 [sys.catalog]: This master's current role is: LEADER
I20260929 18:37:35.320784 16830 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e3f1d175f7240cd9cdfea965742d943 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7e3f1d175f7240cd9cdfea965742d943. Latest consensus state: current_term: 1 leader_uuid: "7e3f1d175f7240cd9cdfea965742d943" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e3f1d175f7240cd9cdfea965742d943" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 33159 } } }
I20260929 18:37:35.320904 16830 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e3f1d175f7240cd9cdfea965742d943 [sys.catalog]: This master's current role is: LEADER
W20260929 18:37:35.323570 16842 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7e3f1d175f7240cd9cdfea965742d943: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:37:35.323633 16842 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:37:35.323710 16843 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:37:35.324534 16843 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:37:35.329838 16843 catalog_manager.cc:1347] Generated new cluster ID: 74a615a1e9474ce38fa89be6a5dd9613
I20260929 18:37:35.329916 16843 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:37:35.341053 16843 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:37:35.342011 16843 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:37:35.352082 16843 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7e3f1d175f7240cd9cdfea965742d943: Generated new TSK 0
I20260929 18:37:35.352751 16843 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.100.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.100.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.100.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.1.keytab.
Entry for principal kudu/127.13.100.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.1.keytab.
I20260929 18:37:35.385684 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707036041078-13712-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707036041078-13712-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707036041078-13712-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707036041078-13712-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.100.1:0
--local_ip_for_outbound_sockets=127.13.100.1
--webserver_interface=127.13.100.1
--webserver_port=0
--tserver_master_addrs=127.13.100.62:33159
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.1.keytab
--principal=kudu/127.13.100.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.100.20:42491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707036041078-13712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707036041078-13712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707036041078-13712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:37:35.511366 16849 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:37:35.511624 16849 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:37:35.511667 16849 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:37:35.515775 16849 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:37:35.515829 16849 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:37:35.515918 16849 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.100.1
I20260929 18:37:35.521211 16849 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.100.20:42491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707036041078-13712-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707036041078-13712-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.1.keytab
--principal=kudu/127.13.100.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.100.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707036041078-13712-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.100.1
--webserver_port=0
--tserver_master_addrs=127.13.100.62:33159
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16849
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.100.1
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707036041078-13712-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:37:35.522318 16849 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:37:35.523312 16849 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:37:35.532529 16854 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:37:35.532559 16855 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:37:35.532697 16857 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:37:35.532763 16849 server_base.cc:1034] running on GCE node
I20260929 18:37:35.533165 16849 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:37:35.533763 16849 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:37:35.534960 16849 hybrid_clock.cc:648] HybridClock initialized: now 1790707055534941 us; error 29 us; skew 500 ppm
Sep 29 18:37:35 dist-test-slave-j1xs krb5kdc[16742](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707055, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:37:35.537720 16849 init.cc:377] Logged in from keytab as kudu/127.13.100.1@KRBTEST.COM (short username kudu)
I20260929 18:37:35.538681 16849 webserver.cc:460] Webserver started at http://127.13.100.1:38523/ using document root <none> and password file <none>
I20260929 18:37:35.539259 16849 fs_manager.cc:359] Metadata directory not provided
I20260929 18:37:35.539354 16849 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:37:35.539588 16849 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:37:35.541615 16849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707036041078-13712-0/minicluster-data/ts-0/data/instance:
uuid: "bd7c36231878412b9d36db1343ef9865"
format_stamp: "Formatted at 2026-09-29 18:37:35 on dist-test-slave-j1xs"
server_key: "2c018bf98ea39ae55e478a4a2896a897"
server_key_iv: "1977362237f3686acf6289af49604c29"
server_key_version: "encryptionkey@0"
I20260929 18:37:35.542241 16849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707036041078-13712-0/minicluster-data/ts-0/wal/instance:
uuid: "bd7c36231878412b9d36db1343ef9865"
format_stamp: "Formatted at 2026-09-29 18:37:35 on dist-test-slave-j1xs"
server_key: "2c018bf98ea39ae55e478a4a2896a897"
server_key_iv: "1977362237f3686acf6289af49604c29"
server_key_version: "encryptionkey@0"
I20260929 18:37:35.546056 16849 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 18:37:35.548473 16864 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:37:35.549656 16849 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:37:35.549796 16849 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707036041078-13712-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707036041078-13712-0/minicluster-data/ts-0/wal
uuid: "bd7c36231878412b9d36db1343ef9865"
format_stamp: "Formatted at 2026-09-29 18:37:35 on dist-test-slave-j1xs"
server_key: "2c018bf98ea39ae55e478a4a2896a897"
server_key_iv: "1977362237f3686acf6289af49604c29"
server_key_version: "encryptionkey@0"
I20260929 18:37:35.549957 16849 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707036041078-13712-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707036041078-13712-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707036041078-13712-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:37:35.568612 16849 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:37:35.571749 16849 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:37:35.571956 16849 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:37:35.572564 16849 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:37:35.573403 16849 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:37:35.573452 16849 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:37:35.573491 16849 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:37:35.573508 16849 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:37:35.582075 16849 rpc_server.cc:307] RPC server started. Bound to: 127.13.100.1:46009
I20260929 18:37:35.582106 16977 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.100.1:46009 every 8 connection(s)
I20260929 18:37:35.582932 16849 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707036041078-13712-0/minicluster-data/ts-0/data/info.pb
I20260929 18:37:35.583251 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 16849
I20260929 18:37:35.583392 13712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707036041078-13712-0/minicluster-data/ts-0/wal/instance
I20260929 18:37:35.583698 13712 external_mini_cluster.cc:1428] Setting key 062ba1d3a489b0cf746da06002bc82bd
Sep 29 18:37:35 dist-test-slave-j1xs krb5kdc[16742](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707055, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.1@KRBTEST.COM for kudu/127.13.100.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.13.100.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.100.2@KRBTEST.COM" created.
I20260929 18:37:35.599015 16826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:35.584770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.1:51509 (local address 127.13.100.62:33159)
0929 18:37:35.584912 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:35.584916 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:35.585860 (+ 944us) server_negotiation.cc:407] Connection header received
0929 18:37:35.587537 (+ 1677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:35.587540 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:35.587583 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:35.587655 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:35.588864 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:35.589414 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:35.591364 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:35.591598 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:35.591688 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:35.594845 (+ 3157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:35.594852 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:35.594855 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:37:35.594891 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:35.596506 (+ 1615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:35.597122 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:35.597128 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:35.597130 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:35.597232 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:35.597667 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:35.597671 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:35.597673 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:35.597925 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:35.598056 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:35.598779 (+ 723us) server_negotiation.cc:299] Negotiation successful
0929 18:37:35.598889 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 18:37:35.600123 16980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:35.585245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:33159 (local address 127.13.100.1:51509)
0929 18:37:35.585689 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:35.585739 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:35.587305 (+ 1566us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:35.587839 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:35.587849 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:35.588240 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:37:35.588715 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:35.588728 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:35.589556 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:35.589562 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:35.591182 (+ 1620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:35.591197 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:35.591809 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:35.591817 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:35.592066 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:35.592754 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:37:35.592782 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:37:35.594654 (+ 1872us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:37:35.596655 (+ 2001us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:35.596662 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:35.596678 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:35.596971 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:35.597335 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:35.597340 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:35.597342 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:35.597494 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:35.598051 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:37:35.598059 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:37:35.598451 (+ 392us) client_negotiation.cc:769] Sending connection context
0929 18:37:35.598678 (+ 227us) client_negotiation.cc:240] Negotiation successful
0929 18:37:35.599061 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":134,"threads_started":1}
I20260929 18:37:35.601548 16978 heartbeater.cc:347] Connected to a master server at 127.13.100.62:33159
I20260929 18:37:35.601817 16978 heartbeater.cc:464] Registering TS with master...
I20260929 18:37:35.602391 16978 heartbeater.cc:511] Master 127.13.100.62:33159 requested a full tablet report, sending...
I20260929 18:37:35.603983 16787 ts_manager.cc:194] Registered new tserver with Master: bd7c36231878412b9d36db1343ef9865 (127.13.100.1:46009)
I20260929 18:37:35.605185 16787 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.100.1@KRBTEST.COM'} at 127.13.100.1:51509
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.100.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.2.keytab.
Entry for principal kudu/127.13.100.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.2.keytab.
I20260929 18:37:35.614954 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707036041078-13712-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707036041078-13712-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707036041078-13712-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707036041078-13712-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.100.2:0
--local_ip_for_outbound_sockets=127.13.100.2
--webserver_interface=127.13.100.2
--webserver_port=0
--tserver_master_addrs=127.13.100.62:33159
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.2.keytab
--principal=kudu/127.13.100.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.100.20:42491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707036041078-13712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707036041078-13712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707036041078-13712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:37:35.726096 16983 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:37:35.726389 16983 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:37:35.726471 16983 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:37:35.730780 16983 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:37:35.730854 16983 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:37:35.730938 16983 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.100.2
I20260929 18:37:35.736124 16983 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.100.20:42491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707036041078-13712-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707036041078-13712-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.2.keytab
--principal=kudu/127.13.100.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.100.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707036041078-13712-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.100.2
--webserver_port=0
--tserver_master_addrs=127.13.100.62:33159
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16983
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.100.2
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707036041078-13712-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:37:35.737119 16983 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:37:35.738162 16983 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:37:35.748103 16988 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:37:35.748107 16991 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:37:35.748077 16989 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:37:35.748245 16983 server_base.cc:1034] running on GCE node
I20260929 18:37:35.748678 16983 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:37:35.749302 16983 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:37:35.750519 16983 hybrid_clock.cc:648] HybridClock initialized: now 1790707055750494 us; error 27 us; skew 500 ppm
Sep 29 18:37:35 dist-test-slave-j1xs krb5kdc[16742](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707055, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:37:35.753278 16983 init.cc:377] Logged in from keytab as kudu/127.13.100.2@KRBTEST.COM (short username kudu)
I20260929 18:37:35.754449 16983 webserver.cc:460] Webserver started at http://127.13.100.2:42299/ using document root <none> and password file <none>
I20260929 18:37:35.755039 16983 fs_manager.cc:359] Metadata directory not provided
I20260929 18:37:35.755131 16983 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:37:35.755334 16983 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:37:35.757294 16983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707036041078-13712-0/minicluster-data/ts-1/data/instance:
uuid: "678c378687454e15bea4bfe786ede9f2"
format_stamp: "Formatted at 2026-09-29 18:37:35 on dist-test-slave-j1xs"
server_key: "f35ce950a2e947ba719d3ccce3c13db0"
server_key_iv: "cb809b4564b92577828600fb326f0f96"
server_key_version: "encryptionkey@0"
I20260929 18:37:35.757848 16983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707036041078-13712-0/minicluster-data/ts-1/wal/instance:
uuid: "678c378687454e15bea4bfe786ede9f2"
format_stamp: "Formatted at 2026-09-29 18:37:35 on dist-test-slave-j1xs"
server_key: "f35ce950a2e947ba719d3ccce3c13db0"
server_key_iv: "cb809b4564b92577828600fb326f0f96"
server_key_version: "encryptionkey@0"
I20260929 18:37:35.761377 16983 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 18:37:35.763754 16998 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:37:35.764973 16983 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:37:35.765134 16983 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707036041078-13712-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707036041078-13712-0/minicluster-data/ts-1/wal
uuid: "678c378687454e15bea4bfe786ede9f2"
format_stamp: "Formatted at 2026-09-29 18:37:35 on dist-test-slave-j1xs"
server_key: "f35ce950a2e947ba719d3ccce3c13db0"
server_key_iv: "cb809b4564b92577828600fb326f0f96"
server_key_version: "encryptionkey@0"
I20260929 18:37:35.765245 16983 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707036041078-13712-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707036041078-13712-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707036041078-13712-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:37:35.777697 16983 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:37:35.780422 16983 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:37:35.780617 16983 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:37:35.781219 16983 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:37:35.782056 16983 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:37:35.782123 16983 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:37:35.782186 16983 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:37:35.782217 16983 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:37:35.791100 16983 rpc_server.cc:307] RPC server started. Bound to: 127.13.100.2:45211
I20260929 18:37:35.791101 17111 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.100.2:45211 every 8 connection(s)
I20260929 18:37:35.791998 16983 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707036041078-13712-0/minicluster-data/ts-1/data/info.pb
I20260929 18:37:35.801406 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 16983
Sep 29 18:37:35 dist-test-slave-j1xs krb5kdc[16742](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707055, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.2@KRBTEST.COM for kudu/127.13.100.62@KRBTEST.COM
I20260929 18:37:35.801527 13712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707036041078-13712-0/minicluster-data/ts-1/wal/instance
I20260929 18:37:35.801872 13712 external_mini_cluster.cc:1428] Setting key d976c37a88c36d905bb716e6c9eb179a
I20260929 18:37:35.805915 16826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:35.793844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.2:36795 (local address 127.13.100.62:33159)
0929 18:37:35.793996 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:35.793999 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:35.794842 (+ 843us) server_negotiation.cc:407] Connection header received
0929 18:37:35.796161 (+ 1319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:35.796164 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:35.796210 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:35.796287 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:35.797459 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:35.798121 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:35.799265 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:35.799500 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:35.799593 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:35.802250 (+ 2657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:35.802254 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:35.802256 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:37:35.802287 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:35.803831 (+ 1544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:35.804361 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:35.804365 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:35.804367 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:35.804447 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:35.804779 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:35.804784 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:35.804786 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:35.804993 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:35.805099 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:35.805597 (+ 498us) server_negotiation.cc:299] Negotiation successful
0929 18:37:35.805715 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 18:37:35.806993 17114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:35.794239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:33159 (local address 127.13.100.2:36795)
0929 18:37:35.794679 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:35.794721 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:35.795956 (+ 1235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:35.796449 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:35.796455 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:35.796807 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:37:35.797292 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:35.797305 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:35.798249 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:35.798252 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:35.799135 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:35.799145 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:35.799643 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:35.799645 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:35.799782 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:35.800413 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:37:35.800429 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:37:35.802076 (+ 1647us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:37:35.803952 (+ 1876us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:35.803957 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:35.803972 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:35.804244 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:35.804540 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:35.804544 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:35.804547 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:35.804686 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:35.805108 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:37:35.805117 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:37:35.805370 (+ 253us) client_negotiation.cc:769] Sending connection context
0929 18:37:35.805558 (+ 188us) client_negotiation.cc:240] Negotiation successful
0929 18:37:35.805878 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":107,"threads_started":1}
I20260929 18:37:35.808357 17112 heartbeater.cc:347] Connected to a master server at 127.13.100.62:33159
I20260929 18:37:35.808645 17112 heartbeater.cc:464] Registering TS with master...
I20260929 18:37:35.809556 17112 heartbeater.cc:511] Master 127.13.100.62:33159 requested a full tablet report, sending...
I20260929 18:37:35.810823 16787 ts_manager.cc:194] Registered new tserver with Master: 678c378687454e15bea4bfe786ede9f2 (127.13.100.2:45211)
I20260929 18:37:35.811407 16787 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.100.2@KRBTEST.COM'} at 127.13.100.2:36795
WARNING: no policy specified for kudu/127.13.100.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.100.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.100.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.3.keytab.
Entry for principal kudu/127.13.100.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.3.keytab.
I20260929 18:37:35.830662 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707036041078-13712-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707036041078-13712-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707036041078-13712-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707036041078-13712-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.100.3:0
--local_ip_for_outbound_sockets=127.13.100.3
--webserver_interface=127.13.100.3
--webserver_port=0
--tserver_master_addrs=127.13.100.62:33159
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.3.keytab
--principal=kudu/127.13.100.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.100.20:42491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707036041078-13712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707036041078-13712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707036041078-13712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:37:35.938586 17117 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:37:35.938827 17117 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:37:35.938867 17117 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:37:35.942821 17117 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:37:35.942873 17117 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:37:35.942929 17117 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.100.3
I20260929 18:37:35.948175 17117 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.100.20:42491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707036041078-13712-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707036041078-13712-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.3.keytab
--principal=kudu/127.13.100.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.100.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707036041078-13712-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.100.3
--webserver_port=0
--tserver_master_addrs=127.13.100.62:33159
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17117
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.100.3
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707036041078-13712-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:37:35.949126 17117 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:37:35.950105 17117 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:37:35.959684 17122 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:37:35.959641 17123 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:37:35.959774 17117 server_base.cc:1034] running on GCE node
W20260929 18:37:35.959641 17125 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:37:35.960285 17117 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:37:35.960877 17117 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:37:35.962015 17117 hybrid_clock.cc:648] HybridClock initialized: now 1790707055962004 us; error 24 us; skew 500 ppm
Sep 29 18:37:35 dist-test-slave-j1xs krb5kdc[16742](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707055, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:37:35.964617 17117 init.cc:377] Logged in from keytab as kudu/127.13.100.3@KRBTEST.COM (short username kudu)
I20260929 18:37:35.965453 17117 webserver.cc:460] Webserver started at http://127.13.100.3:38671/ using document root <none> and password file <none>
I20260929 18:37:35.965994 17117 fs_manager.cc:359] Metadata directory not provided
I20260929 18:37:35.966058 17117 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:37:35.966226 17117 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:37:35.968104 17117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707036041078-13712-0/minicluster-data/ts-2/data/instance:
uuid: "a9527f46ba834564856091740dfc6469"
format_stamp: "Formatted at 2026-09-29 18:37:35 on dist-test-slave-j1xs"
server_key: "e53ff1a73ff6de642c4b2094f179c1dc"
server_key_iv: "6d0e63f9b3ef0370477b0127ca74760f"
server_key_version: "encryptionkey@0"
I20260929 18:37:35.968731 17117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707036041078-13712-0/minicluster-data/ts-2/wal/instance:
uuid: "a9527f46ba834564856091740dfc6469"
format_stamp: "Formatted at 2026-09-29 18:37:35 on dist-test-slave-j1xs"
server_key: "e53ff1a73ff6de642c4b2094f179c1dc"
server_key_iv: "6d0e63f9b3ef0370477b0127ca74760f"
server_key_version: "encryptionkey@0"
I20260929 18:37:35.972286 17117 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260929 18:37:35.974730 17132 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:37:35.975893 17117 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:37:35.976023 17117 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707036041078-13712-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707036041078-13712-0/minicluster-data/ts-2/wal
uuid: "a9527f46ba834564856091740dfc6469"
format_stamp: "Formatted at 2026-09-29 18:37:35 on dist-test-slave-j1xs"
server_key: "e53ff1a73ff6de642c4b2094f179c1dc"
server_key_iv: "6d0e63f9b3ef0370477b0127ca74760f"
server_key_version: "encryptionkey@0"
I20260929 18:37:35.976127 17117 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707036041078-13712-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707036041078-13712-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707036041078-13712-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:37:35.990798 17117 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:37:35.993933 17117 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:37:35.994129 17117 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:37:35.994686 17117 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:37:35.995442 17117 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:37:35.995519 17117 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:37:35.995590 17117 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:37:35.995631 17117 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:37:36.004204 17117 rpc_server.cc:307] RPC server started. Bound to: 127.13.100.3:44853
I20260929 18:37:36.004230 17245 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.100.3:44853 every 8 connection(s)
I20260929 18:37:36.005048 17117 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707036041078-13712-0/minicluster-data/ts-2/data/info.pb
I20260929 18:37:36.006466 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 17117
I20260929 18:37:36.006585 13712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707036041078-13712-0/minicluster-data/ts-2/wal/instance
I20260929 18:37:36.006963 13712 external_mini_cluster.cc:1428] Setting key cf15db8d15dcf44e06610abedb53ebf6
Sep 29 18:37:36 dist-test-slave-j1xs krb5kdc[16742](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707055, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.3@KRBTEST.COM for kudu/127.13.100.62@KRBTEST.COM
I20260929 18:37:36.018163 16826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:36.006828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.3:49355 (local address 127.13.100.62:33159)
0929 18:37:36.006984 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:36.006989 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:36.007998 (+ 1009us) server_negotiation.cc:407] Connection header received
0929 18:37:36.009572 (+ 1574us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:36.009574 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:36.009616 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:36.009683 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:36.010783 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:36.011311 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:36.012353 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:36.012566 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:36.012640 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:36.014978 (+ 2338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:36.014981 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:36.014983 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:37:36.015003 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:36.016466 (+ 1463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:36.016875 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:36.016877 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:36.016879 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:36.016945 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:36.017212 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:36.017214 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:36.017215 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:36.017366 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:36.017437 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:36.017938 (+ 501us) server_negotiation.cc:299] Negotiation successful
0929 18:37:36.018045 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 18:37:36.019115 17248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:36.007286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:33159 (local address 127.13.100.3:49355)
0929 18:37:36.007837 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:36.007917 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:36.009391 (+ 1474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:36.009834 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:36.009841 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:36.010205 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:37:36.010658 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:36.010671 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:36.011412 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:36.011414 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:36.012251 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:36.012260 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:36.012683 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:36.012685 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:36.012814 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:36.013373 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:37:36.013389 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:37:36.014827 (+ 1438us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:37:36.016556 (+ 1729us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:36.016560 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:36.016570 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:36.016794 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:36.017019 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:36.017021 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:36.017022 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:36.017138 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:36.017445 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:37:36.017449 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:37:36.017649 (+ 200us) client_negotiation.cc:769] Sending connection context
0929 18:37:36.017836 (+ 187us) client_negotiation.cc:240] Negotiation successful
0929 18:37:36.018181 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":138,"threads_started":1}
I20260929 18:37:36.020229 17246 heartbeater.cc:347] Connected to a master server at 127.13.100.62:33159
I20260929 18:37:36.020478 17246 heartbeater.cc:464] Registering TS with master...
I20260929 18:37:36.020979 17246 heartbeater.cc:511] Master 127.13.100.62:33159 requested a full tablet report, sending...
I20260929 18:37:36.021876 16782 ts_manager.cc:194] Registered new tserver with Master: a9527f46ba834564856091740dfc6469 (127.13.100.3:44853)
I20260929 18:37:36.022464 16782 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.100.3@KRBTEST.COM'} at 127.13.100.3:49355
I20260929 18:37:36.032474 13712 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 18:37:36 dist-test-slave-j1xs krb5kdc[16742](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707056, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 29 18:37:36 dist-test-slave-j1xs krb5kdc[16742](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707056, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.13.100.62@KRBTEST.COM
I20260929 18:37:36.057142 16826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:36.048889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54138 (local address 127.13.100.62:33159)
0929 18:37:36.049050 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:36.049055 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:36.049126 (+ 71us) server_negotiation.cc:407] Connection header received
0929 18:37:36.049237 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:36.049240 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:36.049281 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:36.049352 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:36.050050 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:36.050555 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:36.051310 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:36.051521 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:36.051598 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:36.054043 (+ 2445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:36.054046 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:36.054048 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:37:36.054071 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:36.055490 (+ 1419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:36.055899 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:36.055904 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:36.055906 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:36.055990 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:36.056255 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:36.056259 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:36.056261 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:36.056484 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:36.056625 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:36.056858 (+ 233us) server_negotiation.cc:299] Negotiation successful
0929 18:37:36.056972 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260929 18:37:36.057621 16782 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:54138
I20260929 18:37:36.058552 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 16849
I20260929 18:37:36.063972 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 16983
I20260929 18:37:36.069149 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 17117
I20260929 18:37:36.074193 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 16755
2026-09-29T18:37:36Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2120 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707036041078-13712-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:37:36 dist-test-slave-j1xs krb5kdc[17258](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:37:36 dist-test-slave-j1xs krb5kdc[17258](info): set up 2 sockets
Sep 29 18:37:36 dist-test-slave-j1xs krb5kdc[17258](info): commencing operation
krb5kdc: starting...
W20260929 18:37:37.126191 13712 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707036041078-13712-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707036041078-13712-0/krb5kdc/test-user.keytab.
Sep 29 18:37:37 dist-test-slave-j1xs krb5kdc[17258](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707057, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:37:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:37:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.100.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.100.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.100.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.62.keytab.
Entry for principal kudu/127.13.100.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.62.keytab.
I20260929 18:37:37.248401 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707036041078-13712-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707036041078-13712-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707036041078-13712-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.100.62:45529
--webserver_interface=127.13.100.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.62.keytab
--principal=kudu/127.13.100.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.100.20:44235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.100.62:45529
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707036041078-13712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707036041078-13712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707036041078-13712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:37:37.355687 17271 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:37:37.355947 17271 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:37:37.355988 17271 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:37:37.359967 17271 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:37:37.360021 17271 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:37:37.360041 17271 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:37:37.360057 17271 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:37:37.360106 17271 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:37:37.365374 17271 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.100.20:44235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707036041078-13712-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707036041078-13712-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.100.62:45529
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.62.keytab
--principal=kudu/127.13.100.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.100.62:45529
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.100.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17271
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707036041078-13712-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:37:37.366437 17271 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:37:37.367426 17271 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:37:37.375842 17279 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:37:37.375819 17277 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:37:37.375990 17271 server_base.cc:1034] running on GCE node
W20260929 18:37:37.375833 17276 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:37:37.376658 17271 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:37:37.377691 17271 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:37:37.378841 17271 hybrid_clock.cc:648] HybridClock initialized: now 1790707057378834 us; error 67 us; skew 500 ppm
Sep 29 18:37:37 dist-test-slave-j1xs krb5kdc[17258](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707057, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:37:37.381649 17271 init.cc:377] Logged in from keytab as kudu/127.13.100.62@KRBTEST.COM (short username kudu)
I20260929 18:37:37.382580 17271 webserver.cc:460] Webserver started at http://127.13.100.62:45257/ using document root <none> and password file <none>
I20260929 18:37:37.383179 17271 fs_manager.cc:359] Metadata directory not provided
I20260929 18:37:37.383244 17271 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:37:37.383457 17271 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:37:37.385525 17271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707036041078-13712-0/minicluster-data/master-0/data/instance:
uuid: "9df14781d15b42f082fdf275a0f9610a"
format_stamp: "Formatted at 2026-09-29 18:37:37 on dist-test-slave-j1xs"
server_key: "451a23a7646f9b9edaeab922fdac8df7"
server_key_iv: "10aa211a039173e763e863afe26f6665"
server_key_version: "encryptionkey@0"
I20260929 18:37:37.386106 17271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707036041078-13712-0/minicluster-data/master-0/wal/instance:
uuid: "9df14781d15b42f082fdf275a0f9610a"
format_stamp: "Formatted at 2026-09-29 18:37:37 on dist-test-slave-j1xs"
server_key: "451a23a7646f9b9edaeab922fdac8df7"
server_key_iv: "10aa211a039173e763e863afe26f6665"
server_key_version: "encryptionkey@0"
I20260929 18:37:37.389655 17271 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260929 18:37:37.392063 17286 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:37:37.393404 17271 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 18:37:37.393555 17271 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707036041078-13712-0/minicluster-data/master-0/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707036041078-13712-0/minicluster-data/master-0/wal
uuid: "9df14781d15b42f082fdf275a0f9610a"
format_stamp: "Formatted at 2026-09-29 18:37:37 on dist-test-slave-j1xs"
server_key: "451a23a7646f9b9edaeab922fdac8df7"
server_key_iv: "10aa211a039173e763e863afe26f6665"
server_key_version: "encryptionkey@0"
I20260929 18:37:37.393664 17271 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707036041078-13712-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707036041078-13712-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707036041078-13712-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:37:37.421090 17271 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:37:37.424510 17271 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:37:37.424667 17271 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:37:37.431921 17271 rpc_server.cc:307] RPC server started. Bound to: 127.13.100.62:45529
I20260929 18:37:37.431943 17338 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.100.62:45529 every 8 connection(s)
I20260929 18:37:37.432910 17271 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
I20260929 18:37:37.434609 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 17271
I20260929 18:37:37.434736 13712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707036041078-13712-0/minicluster-data/master-0/wal/instance
I20260929 18:37:37.435021 13712 external_mini_cluster.cc:1428] Setting key 6f30098d4e45b1b4f0c09308d786a7dd
I20260929 18:37:37.435877 17339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:37:37.442459 17339 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9df14781d15b42f082fdf275a0f9610a: Bootstrap starting.
Sep 29 18:37:37 dist-test-slave-j1xs krb5kdc[17258](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707057, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.100.62@KRBTEST.COM
I20260929 18:37:37.444960 17339 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9df14781d15b42f082fdf275a0f9610a: Neither blocks nor log segments found. Creating new log.
I20260929 18:37:37.445797 17339 log.cc:824] T 00000000000000000000000000000000 P 9df14781d15b42f082fdf275a0f9610a: Log is configured to *not* fsync() on all Append() calls
I20260929 18:37:37.448204 17339 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9df14781d15b42f082fdf275a0f9610a: No bootstrap required, opened a new log
I20260929 18:37:37.450785 17342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:37.436512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40600 (local address 127.13.100.62:45529)
0929 18:37:37.437064 (+ 552us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:37.437085 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:37.437149 (+ 64us) server_negotiation.cc:407] Connection header received
0929 18:37:37.438221 (+ 1072us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:37.438247 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:37.438668 (+ 421us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:37.439029 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:37.440210 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:37.441210 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:37.442133 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:37.442639 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:37.442794 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:37.445479 (+ 2685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:37.445486 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:37.445502 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:37:37.445532 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:37.447652 (+ 2120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:37.448088 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:37.448093 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:37.448099 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:37.448227 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:37.448566 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:37.448571 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:37.448574 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:37.449048 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:37.449226 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:37.449443 (+ 217us) server_negotiation.cc:299] Negotiation successful
0929 18:37:37.449728 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":139,"threads_started":1}
I20260929 18:37:37.452620 17339 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9df14781d15b42f082fdf275a0f9610a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9df14781d15b42f082fdf275a0f9610a" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 45529 } }
I20260929 18:37:37.452919 17339 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9df14781d15b42f082fdf275a0f9610a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:37:37.453012 17339 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9df14781d15b42f082fdf275a0f9610a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9df14781d15b42f082fdf275a0f9610a, State: Initialized, Role: FOLLOWER
I20260929 18:37:37.453540 17339 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9df14781d15b42f082fdf275a0f9610a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9df14781d15b42f082fdf275a0f9610a" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 45529 } }
I20260929 18:37:37.453691 17339 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9df14781d15b42f082fdf275a0f9610a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:37:37.453733 17339 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9df14781d15b42f082fdf275a0f9610a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:37:37.453804 17339 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9df14781d15b42f082fdf275a0f9610a [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:37:37.454744 17339 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9df14781d15b42f082fdf275a0f9610a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9df14781d15b42f082fdf275a0f9610a" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 45529 } }
I20260929 18:37:37.455037 17339 leader_election.cc:302] T 00000000000000000000000000000000 P 9df14781d15b42f082fdf275a0f9610a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9df14781d15b42f082fdf275a0f9610a; no voters:
I20260929 18:37:37.455289 17339 leader_election.cc:288] T 00000000000000000000000000000000 P 9df14781d15b42f082fdf275a0f9610a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:37:37.455463 17344 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9df14781d15b42f082fdf275a0f9610a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:37:37.455708 17344 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9df14781d15b42f082fdf275a0f9610a [term 1 LEADER]: Becoming Leader. State: Replica: 9df14781d15b42f082fdf275a0f9610a, State: Running, Role: LEADER
I20260929 18:37:37.456143 17344 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9df14781d15b42f082fdf275a0f9610a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9df14781d15b42f082fdf275a0f9610a" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 45529 } }
I20260929 18:37:37.456657 17339 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9df14781d15b42f082fdf275a0f9610a [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:37:37.457764 17346 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9df14781d15b42f082fdf275a0f9610a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9df14781d15b42f082fdf275a0f9610a. Latest consensus state: current_term: 1 leader_uuid: "9df14781d15b42f082fdf275a0f9610a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9df14781d15b42f082fdf275a0f9610a" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 45529 } } }
I20260929 18:37:37.457906 17346 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9df14781d15b42f082fdf275a0f9610a [sys.catalog]: This master's current role is: LEADER
I20260929 18:37:37.458192 17345 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9df14781d15b42f082fdf275a0f9610a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9df14781d15b42f082fdf275a0f9610a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9df14781d15b42f082fdf275a0f9610a" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 45529 } } }
I20260929 18:37:37.458273 17345 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9df14781d15b42f082fdf275a0f9610a [sys.catalog]: This master's current role is: LEADER
I20260929 18:37:37.459126 17353 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:37:37.461714 17353 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:37:37.467080 17353 catalog_manager.cc:1347] Generated new cluster ID: 712be549764f4902b6771094d7e91dd5
I20260929 18:37:37.467149 17353 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:37:37.482618 17353 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:37:37.483582 17353 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:37:37.491288 17353 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9df14781d15b42f082fdf275a0f9610a: Generated new TSK 0
I20260929 18:37:37.491967 17353 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.100.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.100.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.100.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.1.keytab.
Entry for principal kudu/127.13.100.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.1.keytab.
I20260929 18:37:37.524580 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707036041078-13712-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707036041078-13712-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707036041078-13712-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707036041078-13712-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.100.1:0
--local_ip_for_outbound_sockets=127.13.100.1
--webserver_interface=127.13.100.1
--webserver_port=0
--tserver_master_addrs=127.13.100.62:45529
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.1.keytab
--principal=kudu/127.13.100.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.100.20:44235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707036041078-13712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707036041078-13712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707036041078-13712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:37:37.660514 17365 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:37:37.660782 17365 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:37:37.660866 17365 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:37:37.664938 17365 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:37:37.665007 17365 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:37:37.665103 17365 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.100.1
I20260929 18:37:37.670796 17365 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.100.20:44235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707036041078-13712-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707036041078-13712-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.1.keytab
--principal=kudu/127.13.100.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.100.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707036041078-13712-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.100.1
--webserver_port=0
--tserver_master_addrs=127.13.100.62:45529
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17365
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.100.1
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707036041078-13712-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:37:37.671905 17365 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:37:37.672943 17365 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:37:37.682368 17370 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:37:37.682539 17371 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:37:37.682358 17373 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:37:37.682725 17365 server_base.cc:1034] running on GCE node
I20260929 18:37:37.683097 17365 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:37:37.683740 17365 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:37:37.684906 17365 hybrid_clock.cc:648] HybridClock initialized: now 1790707057684888 us; error 31 us; skew 500 ppm
Sep 29 18:37:37 dist-test-slave-j1xs krb5kdc[17258](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707057, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:37:37.687533 17365 init.cc:377] Logged in from keytab as kudu/127.13.100.1@KRBTEST.COM (short username kudu)
I20260929 18:37:37.688336 17365 webserver.cc:460] Webserver started at http://127.13.100.1:38621/ using document root <none> and password file <none>
I20260929 18:37:37.688897 17365 fs_manager.cc:359] Metadata directory not provided
I20260929 18:37:37.688992 17365 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:37:37.689235 17365 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:37:37.691252 17365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707036041078-13712-0/minicluster-data/ts-0/data/instance:
uuid: "b49910893a3d45799aa9716ad973e523"
format_stamp: "Formatted at 2026-09-29 18:37:37 on dist-test-slave-j1xs"
server_key: "59699c04f27b4ee85ce2003b60820110"
server_key_iv: "10c409773972bd3246e200e8d7ad73dd"
server_key_version: "encryptionkey@0"
I20260929 18:37:37.691790 17365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707036041078-13712-0/minicluster-data/ts-0/wal/instance:
uuid: "b49910893a3d45799aa9716ad973e523"
format_stamp: "Formatted at 2026-09-29 18:37:37 on dist-test-slave-j1xs"
server_key: "59699c04f27b4ee85ce2003b60820110"
server_key_iv: "10c409773972bd3246e200e8d7ad73dd"
server_key_version: "encryptionkey@0"
I20260929 18:37:37.695339 17365 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 18:37:37.697743 17380 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:37:37.698994 17365 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:37:37.699117 17365 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707036041078-13712-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707036041078-13712-0/minicluster-data/ts-0/wal
uuid: "b49910893a3d45799aa9716ad973e523"
format_stamp: "Formatted at 2026-09-29 18:37:37 on dist-test-slave-j1xs"
server_key: "59699c04f27b4ee85ce2003b60820110"
server_key_iv: "10c409773972bd3246e200e8d7ad73dd"
server_key_version: "encryptionkey@0"
I20260929 18:37:37.699231 17365 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707036041078-13712-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707036041078-13712-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707036041078-13712-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:37:37.729372 17365 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:37:37.732547 17365 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:37:37.732761 17365 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:37:37.733376 17365 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:37:37.734288 17365 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:37:37.734359 17365 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:37:37.734431 17365 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:37:37.734476 17365 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:37:37.743322 17365 rpc_server.cc:307] RPC server started. Bound to: 127.13.100.1:39879
I20260929 18:37:37.743340 17493 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.100.1:39879 every 8 connection(s)
I20260929 18:37:37.744234 17365 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707036041078-13712-0/minicluster-data/ts-0/data/info.pb
I20260929 18:37:37.751922 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 17365
I20260929 18:37:37.752074 13712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707036041078-13712-0/minicluster-data/ts-0/wal/instance
I20260929 18:37:37.752351 13712 external_mini_cluster.cc:1428] Setting key 7343b62ed85164c276c82a114aa82b3a
Sep 29 18:37:37 dist-test-slave-j1xs krb5kdc[17258](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707057, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.1@KRBTEST.COM for kudu/127.13.100.62@KRBTEST.COM
I20260929 18:37:37.758387 17342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:37.745969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.1:44621 (local address 127.13.100.62:45529)
0929 18:37:37.746099 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:37.746103 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:37.746894 (+ 791us) server_negotiation.cc:407] Connection header received
0929 18:37:37.748288 (+ 1394us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:37.748291 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:37.748347 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:37.748430 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:37.749590 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:37.750206 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:37.751326 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:37.751564 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:37.751677 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:37.754491 (+ 2814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:37.754495 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:37.754497 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:37:37.754520 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:37.756218 (+ 1698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:37.756740 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:37.756746 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:37.756751 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:37.756843 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:37.757164 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:37.757170 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:37.757175 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:37.757415 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:37.757526 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:37.758042 (+ 516us) server_negotiation.cc:299] Negotiation successful
0929 18:37:37.758191 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 18:37:37.759160 17496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:37.746336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:45529 (local address 127.13.100.1:44621)
0929 18:37:37.746756 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:37.746796 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:37.748077 (+ 1281us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:37.748627 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:37.748634 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:37.748976 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:37:37.749428 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:37.749456 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:37.750335 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:37.750339 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:37.751200 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:37.751209 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:37.751715 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:37.751719 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:37.751879 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:37.752538 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:37:37.752554 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:37:37.754332 (+ 1778us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:37:37.756376 (+ 2044us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:37.756381 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:37.756393 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:37.756641 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:37.756935 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:37.756938 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:37.756939 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:37.757075 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:37.757519 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:37:37.757524 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:37:37.757748 (+ 224us) client_negotiation.cc:769] Sending connection context
0929 18:37:37.758006 (+ 258us) client_negotiation.cc:240] Negotiation successful
0929 18:37:37.758275 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":114,"threads_started":1}
I20260929 18:37:37.760257 17494 heartbeater.cc:347] Connected to a master server at 127.13.100.62:45529
I20260929 18:37:37.760515 17494 heartbeater.cc:464] Registering TS with master...
I20260929 18:37:37.761104 17494 heartbeater.cc:511] Master 127.13.100.62:45529 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.13.100.2@KRBTEST.COM; defaulting to no policy
I20260929 18:37:37.762708 17303 ts_manager.cc:194] Registered new tserver with Master: b49910893a3d45799aa9716ad973e523 (127.13.100.1:39879)
I20260929 18:37:37.763877 17303 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.100.1@KRBTEST.COM'} at 127.13.100.1:44621
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.100.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.100.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.2.keytab.
Entry for principal kudu/127.13.100.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.2.keytab.
I20260929 18:37:37.780998 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707036041078-13712-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707036041078-13712-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707036041078-13712-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707036041078-13712-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.100.2:0
--local_ip_for_outbound_sockets=127.13.100.2
--webserver_interface=127.13.100.2
--webserver_port=0
--tserver_master_addrs=127.13.100.62:45529
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.2.keytab
--principal=kudu/127.13.100.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.100.20:44235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707036041078-13712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707036041078-13712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707036041078-13712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:37:37.888120 17499 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:37:37.888338 17499 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:37:37.888378 17499 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:37:37.892519 17499 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:37:37.892566 17499 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:37:37.892621 17499 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.100.2
I20260929 18:37:37.897616 17499 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.100.20:44235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707036041078-13712-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707036041078-13712-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.2.keytab
--principal=kudu/127.13.100.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.100.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707036041078-13712-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.100.2
--webserver_port=0
--tserver_master_addrs=127.13.100.62:45529
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17499
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.100.2
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707036041078-13712-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:37:37.898792 17499 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:37:37.899730 17499 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:37:37.909240 17505 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:37:37.909235 17504 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:37:37.909430 17499 server_base.cc:1034] running on GCE node
W20260929 18:37:37.909238 17507 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:37:37.909972 17499 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:37:37.910616 17499 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:37:37.911785 17499 hybrid_clock.cc:648] HybridClock initialized: now 1790707057911751 us; error 45 us; skew 500 ppm
Sep 29 18:37:37 dist-test-slave-j1xs krb5kdc[17258](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707057, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:37:37.914394 17499 init.cc:377] Logged in from keytab as kudu/127.13.100.2@KRBTEST.COM (short username kudu)
I20260929 18:37:37.915235 17499 webserver.cc:460] Webserver started at http://127.13.100.2:46447/ using document root <none> and password file <none>
I20260929 18:37:37.915717 17499 fs_manager.cc:359] Metadata directory not provided
I20260929 18:37:37.915773 17499 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:37:37.915930 17499 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:37:37.918049 17499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707036041078-13712-0/minicluster-data/ts-1/data/instance:
uuid: "5f01f1b44e4a47f3b5f5396f7ade3a6e"
format_stamp: "Formatted at 2026-09-29 18:37:37 on dist-test-slave-j1xs"
server_key: "69300f09b37b41efd20e0d3b77d6dd8e"
server_key_iv: "c037d3eaa3b2f9a9d86c289100b71c57"
server_key_version: "encryptionkey@0"
I20260929 18:37:37.918533 17499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707036041078-13712-0/minicluster-data/ts-1/wal/instance:
uuid: "5f01f1b44e4a47f3b5f5396f7ade3a6e"
format_stamp: "Formatted at 2026-09-29 18:37:37 on dist-test-slave-j1xs"
server_key: "69300f09b37b41efd20e0d3b77d6dd8e"
server_key_iv: "c037d3eaa3b2f9a9d86c289100b71c57"
server_key_version: "encryptionkey@0"
I20260929 18:37:37.921839 17499 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:37:37.924139 17514 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:37:37.925292 17499 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:37:37.925515 17499 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707036041078-13712-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707036041078-13712-0/minicluster-data/ts-1/wal
uuid: "5f01f1b44e4a47f3b5f5396f7ade3a6e"
format_stamp: "Formatted at 2026-09-29 18:37:37 on dist-test-slave-j1xs"
server_key: "69300f09b37b41efd20e0d3b77d6dd8e"
server_key_iv: "c037d3eaa3b2f9a9d86c289100b71c57"
server_key_version: "encryptionkey@0"
I20260929 18:37:37.925621 17499 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707036041078-13712-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707036041078-13712-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707036041078-13712-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:37:37.940160 17499 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:37:37.942971 17499 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:37:37.943188 17499 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:37:37.943739 17499 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:37:37.944506 17499 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:37:37.944576 17499 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:37:37.944643 17499 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:37:37.944679 17499 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:37:37.953228 17499 rpc_server.cc:307] RPC server started. Bound to: 127.13.100.2:36465
I20260929 18:37:37.953250 17627 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.100.2:36465 every 8 connection(s)
I20260929 18:37:37.954110 17499 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707036041078-13712-0/minicluster-data/ts-1/data/info.pb
I20260929 18:37:37.956887 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 17499
I20260929 18:37:37.957005 13712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707036041078-13712-0/minicluster-data/ts-1/wal/instance
I20260929 18:37:37.957309 13712 external_mini_cluster.cc:1428] Setting key 431a252399516bc5f82427115dfcf7a4
Sep 29 18:37:37 dist-test-slave-j1xs krb5kdc[17258](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707057, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.2@KRBTEST.COM for kudu/127.13.100.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.13.100.3@KRBTEST.COM; defaulting to no policy
I20260929 18:37:37.969123 17342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:37.955868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.2:36921 (local address 127.13.100.62:45529)
0929 18:37:37.956006 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:37.956011 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:37.956967 (+ 956us) server_negotiation.cc:407] Connection header received
0929 18:37:37.958694 (+ 1727us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:37.958699 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:37.958747 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:37.958852 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:37.960436 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:37.961307 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:37.962479 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:37.962902 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:37.963025 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:37.965587 (+ 2562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:37.965593 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:37.965598 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:37:37.965629 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:37.967242 (+ 1613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:37.967738 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:37.967741 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:37.967742 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:37.967812 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:37.968099 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:37.968101 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:37.968102 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:37.968260 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:37.968366 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:37.968910 (+ 544us) server_negotiation.cc:299] Negotiation successful
0929 18:37:37.969005 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.100.3@KRBTEST.COM" created.
I20260929 18:37:37.970028 17630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:37.956303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:45529 (local address 127.13.100.2:36921)
0929 18:37:37.956802 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:37.956861 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:37.958504 (+ 1643us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:37.959093 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:37.959102 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:37.959598 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:37:37.960256 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:37.960278 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:37.961437 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:37.961439 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:37.962361 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:37.962370 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:37.963069 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:37.963073 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:37.963226 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:37.963806 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:37:37.963822 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:37:37.965423 (+ 1601us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:37:37.967378 (+ 1955us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:37.967390 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:37.967401 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:37.967645 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:37.967891 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:37.967893 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:37.967894 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:37.968014 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:37.968359 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:37:37.968364 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:37:37.968666 (+ 302us) client_negotiation.cc:769] Sending connection context
0929 18:37:37.968818 (+ 152us) client_negotiation.cc:240] Negotiation successful
0929 18:37:37.969060 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":110,"threads_started":1}
I20260929 18:37:37.971161 17628 heartbeater.cc:347] Connected to a master server at 127.13.100.62:45529
I20260929 18:37:37.971395 17628 heartbeater.cc:464] Registering TS with master...
I20260929 18:37:37.971864 17628 heartbeater.cc:511] Master 127.13.100.62:45529 requested a full tablet report, sending...
I20260929 18:37:37.972887 17303 ts_manager.cc:194] Registered new tserver with Master: 5f01f1b44e4a47f3b5f5396f7ade3a6e (127.13.100.2:36465)
I20260929 18:37:37.973727 17303 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.100.2@KRBTEST.COM'} at 127.13.100.2:36921
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.100.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.3.keytab.
Entry for principal kudu/127.13.100.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.3.keytab.
I20260929 18:37:37.986011 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707036041078-13712-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707036041078-13712-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707036041078-13712-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707036041078-13712-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.100.3:0
--local_ip_for_outbound_sockets=127.13.100.3
--webserver_interface=127.13.100.3
--webserver_port=0
--tserver_master_addrs=127.13.100.62:45529
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.3.keytab
--principal=kudu/127.13.100.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.100.20:44235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707036041078-13712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707036041078-13712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707036041078-13712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:37:38.093621 17633 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:37:38.093919 17633 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:37:38.094031 17633 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:37:38.098363 17633 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:37:38.098430 17633 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:37:38.098492 17633 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.100.3
I20260929 18:37:38.103641 17633 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.100.20:44235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707036041078-13712-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707036041078-13712-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.3.keytab
--principal=kudu/127.13.100.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.100.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707036041078-13712-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.100.3
--webserver_port=0
--tserver_master_addrs=127.13.100.62:45529
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17633
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.100.3
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707036041078-13712-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:37:38.104606 17633 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:37:38.105569 17633 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:37:38.115185 17639 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:37:38.115249 17641 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:37:38.115199 17638 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:37:38.115270 17633 server_base.cc:1034] running on GCE node
I20260929 18:37:38.115888 17633 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:37:38.116555 17633 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:37:38.117710 17633 hybrid_clock.cc:648] HybridClock initialized: now 1790707058117699 us; error 27 us; skew 500 ppm
Sep 29 18:37:38 dist-test-slave-j1xs krb5kdc[17258](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707058, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:37:38.120433 17633 init.cc:377] Logged in from keytab as kudu/127.13.100.3@KRBTEST.COM (short username kudu)
I20260929 18:37:38.121330 17633 webserver.cc:460] Webserver started at http://127.13.100.3:46149/ using document root <none> and password file <none>
I20260929 18:37:38.121946 17633 fs_manager.cc:359] Metadata directory not provided
I20260929 18:37:38.122031 17633 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:37:38.122231 17633 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:37:38.124202 17633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707036041078-13712-0/minicluster-data/ts-2/data/instance:
uuid: "193f7052e59944a4b7d98b7ea3fc0bb9"
format_stamp: "Formatted at 2026-09-29 18:37:38 on dist-test-slave-j1xs"
server_key: "7e1e71af6ef251c931c6eed11edf5097"
server_key_iv: "819cd962152a0a6d86cc6d8182ac15b6"
server_key_version: "encryptionkey@0"
I20260929 18:37:38.124748 17633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707036041078-13712-0/minicluster-data/ts-2/wal/instance:
uuid: "193f7052e59944a4b7d98b7ea3fc0bb9"
format_stamp: "Formatted at 2026-09-29 18:37:38 on dist-test-slave-j1xs"
server_key: "7e1e71af6ef251c931c6eed11edf5097"
server_key_iv: "819cd962152a0a6d86cc6d8182ac15b6"
server_key_version: "encryptionkey@0"
I20260929 18:37:38.128407 17633 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260929 18:37:38.130893 17648 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:37:38.132047 17633 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 18:37:38.132184 17633 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707036041078-13712-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707036041078-13712-0/minicluster-data/ts-2/wal
uuid: "193f7052e59944a4b7d98b7ea3fc0bb9"
format_stamp: "Formatted at 2026-09-29 18:37:38 on dist-test-slave-j1xs"
server_key: "7e1e71af6ef251c931c6eed11edf5097"
server_key_iv: "819cd962152a0a6d86cc6d8182ac15b6"
server_key_version: "encryptionkey@0"
I20260929 18:37:38.132290 17633 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707036041078-13712-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707036041078-13712-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707036041078-13712-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:37:38.157743 17633 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:37:38.160799 17633 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:37:38.160960 17633 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:37:38.161545 17633 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:37:38.162366 17633 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:37:38.162413 17633 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:37:38.162452 17633 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:37:38.162467 17633 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:37:38.170858 17633 rpc_server.cc:307] RPC server started. Bound to: 127.13.100.3:34879
I20260929 18:37:38.170997 17761 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.100.3:34879 every 8 connection(s)
I20260929 18:37:38.171794 17633 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707036041078-13712-0/minicluster-data/ts-2/data/info.pb
Sep 29 18:37:38 dist-test-slave-j1xs krb5kdc[17258](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707058, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.3@KRBTEST.COM for kudu/127.13.100.62@KRBTEST.COM
I20260929 18:37:38.181936 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 17633
I20260929 18:37:38.182072 13712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707036041078-13712-0/minicluster-data/ts-2/wal/instance
I20260929 18:37:38.182366 13712 external_mini_cluster.cc:1428] Setting key 54345b8544d87be31becc4fb34f57abd
I20260929 18:37:38.185451 17342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:38.173542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.3:43701 (local address 127.13.100.62:45529)
0929 18:37:38.173700 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:38.173705 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:38.174509 (+ 804us) server_negotiation.cc:407] Connection header received
0929 18:37:38.175798 (+ 1289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:38.175801 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:38.175845 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:38.175952 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:38.177152 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:38.177670 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:38.178784 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:38.178995 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:38.179086 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:38.181354 (+ 2268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:38.181358 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:38.181360 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:37:38.181382 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:38.183028 (+ 1646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:38.183523 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:38.183527 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:38.183528 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:38.183619 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:38.184424 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:38.184428 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:38.184430 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:38.184615 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:38.184701 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:38.185195 (+ 494us) server_negotiation.cc:299] Negotiation successful
0929 18:37:38.185319 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 18:37:38.186173 17764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:38.173955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:45529 (local address 127.13.100.3:43701)
0929 18:37:38.174364 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:38.174404 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:38.175608 (+ 1204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:38.176110 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:38.176117 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:38.176449 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:37:38.176992 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:38.177012 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:38.177771 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:38.177773 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:38.178674 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:38.178682 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:38.179129 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:38.179131 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:38.179261 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:38.179790 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:37:38.179805 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:37:38.181180 (+ 1375us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:37:38.183171 (+ 1991us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:38.183179 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:38.183194 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:38.183444 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:38.184202 (+ 758us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:38.184205 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:38.184206 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:38.184343 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:38.184708 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:37:38.184713 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:37:38.184915 (+ 202us) client_negotiation.cc:769] Sending connection context
0929 18:37:38.185078 (+ 163us) client_negotiation.cc:240] Negotiation successful
0929 18:37:38.185320 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":111,"threads_started":1}
I20260929 18:37:38.187379 17762 heartbeater.cc:347] Connected to a master server at 127.13.100.62:45529
I20260929 18:37:38.187642 17762 heartbeater.cc:464] Registering TS with master...
I20260929 18:37:38.188337 17762 heartbeater.cc:511] Master 127.13.100.62:45529 requested a full tablet report, sending...
I20260929 18:37:38.189396 17303 ts_manager.cc:194] Registered new tserver with Master: 193f7052e59944a4b7d98b7ea3fc0bb9 (127.13.100.3:34879)
I20260929 18:37:38.190045 17303 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.100.3@KRBTEST.COM'} at 127.13.100.3:43701
I20260929 18:37:38.196017 13712 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 18:37:38 dist-test-slave-j1xs krb5kdc[17258](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707058, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 18:37:38 dist-test-slave-j1xs krb5kdc[17258](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707058, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.100.1@KRBTEST.COM
I20260929 18:37:38.222046 17769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:38.212847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45724 (local address 127.13.100.1:39879)
0929 18:37:38.213197 (+ 350us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:38.213211 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:38.213233 (+ 22us) server_negotiation.cc:407] Connection header received
0929 18:37:38.213280 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:38.213285 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:38.213468 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:38.213587 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:38.214388 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:38.215152 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:38.215931 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:38.216279 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:38.216356 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:38.218645 (+ 2289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:38.218653 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:38.218666 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:37:38.218689 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:38.220319 (+ 1630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:38.220713 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:38.220718 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:38.220720 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:38.220831 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:38.221146 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:38.221148 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:38.221149 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:38.221369 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:38.221520 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:38.221692 (+ 172us) server_negotiation.cc:299] Negotiation successful
0929 18:37:38.221811 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":157,"threads_started":1}
Sep 29 18:37:38 dist-test-slave-j1xs krb5kdc[17258](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707058, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.100.62@KRBTEST.COM
I20260929 18:37:38.232515 17342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:38.224798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40610 (local address 127.13.100.62:45529)
0929 18:37:38.224933 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:38.224938 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:38.224978 (+ 40us) server_negotiation.cc:407] Connection header received
0929 18:37:38.225137 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:38.225140 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:38.225191 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:38.225276 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:38.225841 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:38.226397 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:38.227154 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:38.227379 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:38.227492 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:38.229684 (+ 2192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:38.229688 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:38.229690 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:37:38.229717 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:38.230898 (+ 1181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:38.231297 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:38.231302 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:38.231304 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:38.231382 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:38.231660 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:38.231663 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:38.231665 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:38.231866 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:38.232003 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:38.232235 (+ 232us) server_negotiation.cc:299] Negotiation successful
0929 18:37:38.232348 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260929 18:37:38.232931 17303 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40610
I20260929 18:37:38.233541 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 17365
I20260929 18:37:38.239455 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 17499
I20260929 18:37:38.244697 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 17633
I20260929 18:37:38.249747 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 17271
2026-09-29T18:37:38Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2175 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707036041078-13712-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:37:38 dist-test-slave-j1xs krb5kdc[17774](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:37:38 dist-test-slave-j1xs krb5kdc[17774](info): set up 2 sockets
Sep 29 18:37:38 dist-test-slave-j1xs krb5kdc[17774](info): commencing operation
krb5kdc: starting...
W20260929 18:37:39.297976 13712 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707036041078-13712-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707036041078-13712-0/krb5kdc/test-user.keytab.
Sep 29 18:37:39 dist-test-slave-j1xs krb5kdc[17774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707059, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:37:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:37:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.100.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.100.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.100.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.62.keytab.
Entry for principal kudu/127.13.100.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.62.keytab.
I20260929 18:37:39.417335 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707036041078-13712-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707036041078-13712-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707036041078-13712-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.100.62:39601
--webserver_interface=127.13.100.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.62.keytab
--principal=kudu/127.13.100.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.100.20:39039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.100.62:39601
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707036041078-13712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707036041078-13712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707036041078-13712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:37:39.531139 17787 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:37:39.531407 17787 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:37:39.531452 17787 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:37:39.535370 17787 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:37:39.535425 17787 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:37:39.535445 17787 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:37:39.535461 17787 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:37:39.535477 17787 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:37:39.540648 17787 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.100.20:39039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707036041078-13712-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707036041078-13712-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.100.62:39601
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.62.keytab
--principal=kudu/127.13.100.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.100.62:39601
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.100.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17787
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707036041078-13712-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:37:39.541625 17787 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:37:39.542667 17787 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:37:39.551206 17792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:37:39.551226 17793 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:37:39.551344 17787 server_base.cc:1034] running on GCE node
W20260929 18:37:39.551199 17795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:37:39.552024 17787 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:37:39.553135 17787 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:37:39.554322 17787 hybrid_clock.cc:648] HybridClock initialized: now 1790707059554283 us; error 50 us; skew 500 ppm
Sep 29 18:37:39 dist-test-slave-j1xs krb5kdc[17774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707059, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:37:39.557111 17787 init.cc:377] Logged in from keytab as kudu/127.13.100.62@KRBTEST.COM (short username kudu)
I20260929 18:37:39.558046 17787 fs_manager.cc:359] Metadata directory not provided
I20260929 18:37:39.558135 17787 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:37:39.558341 17787 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:37:39.560307 17787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707036041078-13712-0/minicluster-data/master-0/data/instance:
uuid: "dc643846dbc84628ba4ec83ffbc08570"
format_stamp: "Formatted at 2026-09-29 18:37:39 on dist-test-slave-j1xs"
server_key: "6cad21056ddbfd43bc1ccefbe6cf392c"
server_key_iv: "3c015dfa08b5e020b4c6cc4dae25c305"
server_key_version: "encryptionkey@0"
I20260929 18:37:39.560985 17787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707036041078-13712-0/minicluster-data/master-0/wal/instance:
uuid: "dc643846dbc84628ba4ec83ffbc08570"
format_stamp: "Formatted at 2026-09-29 18:37:39 on dist-test-slave-j1xs"
server_key: "6cad21056ddbfd43bc1ccefbe6cf392c"
server_key_iv: "3c015dfa08b5e020b4c6cc4dae25c305"
server_key_version: "encryptionkey@0"
I20260929 18:37:39.564775 17787 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260929 18:37:39.567627 17801 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:37:39.568848 17787 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 18:37:39.568984 17787 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707036041078-13712-0/minicluster-data/master-0/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707036041078-13712-0/minicluster-data/master-0/wal
uuid: "dc643846dbc84628ba4ec83ffbc08570"
format_stamp: "Formatted at 2026-09-29 18:37:39 on dist-test-slave-j1xs"
server_key: "6cad21056ddbfd43bc1ccefbe6cf392c"
server_key_iv: "3c015dfa08b5e020b4c6cc4dae25c305"
server_key_version: "encryptionkey@0"
I20260929 18:37:39.569103 17787 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707036041078-13712-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707036041078-13712-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707036041078-13712-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:37:39.587057 17787 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:37:39.590612 17787 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:37:39.590826 17787 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:37:39.597570 17787 rpc_server.cc:307] RPC server started. Bound to: 127.13.100.62:39601
I20260929 18:37:39.597853 17853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.100.62:39601 every 8 connection(s)
I20260929 18:37:39.598278 17787 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
I20260929 18:37:39.600806 17854 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:37:39.602973 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 17787
I20260929 18:37:39.603111 13712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707036041078-13712-0/minicluster-data/master-0/wal/instance
I20260929 18:37:39.603387 13712 external_mini_cluster.cc:1428] Setting key 46870b2f47f1d7699636e4d1cce51306
I20260929 18:37:39.606920 17854 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc643846dbc84628ba4ec83ffbc08570: Bootstrap starting.
I20260929 18:37:39.609386 17854 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dc643846dbc84628ba4ec83ffbc08570: Neither blocks nor log segments found. Creating new log.
I20260929 18:37:39.610332 17854 log.cc:824] T 00000000000000000000000000000000 P dc643846dbc84628ba4ec83ffbc08570: Log is configured to *not* fsync() on all Append() calls
Sep 29 18:37:39 dist-test-slave-j1xs krb5kdc[17774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707059, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.100.62@KRBTEST.COM
I20260929 18:37:39.612578 17854 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc643846dbc84628ba4ec83ffbc08570: No bootstrap required, opened a new log
I20260929 18:37:39.615548 17854 raft_consensus.cc:348] T 00000000000000000000000000000000 P dc643846dbc84628ba4ec83ffbc08570 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc643846dbc84628ba4ec83ffbc08570" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 39601 } }
I20260929 18:37:39.615770 17854 raft_consensus.cc:374] T 00000000000000000000000000000000 P dc643846dbc84628ba4ec83ffbc08570 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:37:39.615831 17854 raft_consensus.cc:729] T 00000000000000000000000000000000 P dc643846dbc84628ba4ec83ffbc08570 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc643846dbc84628ba4ec83ffbc08570, State: Initialized, Role: FOLLOWER
I20260929 18:37:39.616248 17854 consensus_queue.cc:260] T 00000000000000000000000000000000 P dc643846dbc84628ba4ec83ffbc08570 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc643846dbc84628ba4ec83ffbc08570" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 39601 } }
I20260929 18:37:39.616398 17854 raft_consensus.cc:388] T 00000000000000000000000000000000 P dc643846dbc84628ba4ec83ffbc08570 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:37:39.616483 17854 raft_consensus.cc:482] T 00000000000000000000000000000000 P dc643846dbc84628ba4ec83ffbc08570 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:37:39.616600 17854 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dc643846dbc84628ba4ec83ffbc08570 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:37:39.617928 17854 raft_consensus.cc:504] T 00000000000000000000000000000000 P dc643846dbc84628ba4ec83ffbc08570 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc643846dbc84628ba4ec83ffbc08570" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 39601 } }
I20260929 18:37:39.618283 17854 leader_election.cc:302] T 00000000000000000000000000000000 P dc643846dbc84628ba4ec83ffbc08570 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dc643846dbc84628ba4ec83ffbc08570; no voters:
I20260929 18:37:39.618637 17854 leader_election.cc:288] T 00000000000000000000000000000000 P dc643846dbc84628ba4ec83ffbc08570 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:37:39.618765 17859 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dc643846dbc84628ba4ec83ffbc08570 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:37:39.618952 17857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:39.604697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42400 (local address 127.13.100.62:39601)
0929 18:37:39.605073 (+ 376us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:39.605087 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:39.605120 (+ 33us) server_negotiation.cc:407] Connection header received
0929 18:37:39.605988 (+ 868us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:39.606007 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:39.606323 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:39.606602 (+ 279us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:39.607467 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:39.608479 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:39.609642 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:39.610229 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:39.610350 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:39.613206 (+ 2856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:39.613211 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:39.613224 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:37:39.613250 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:39.615473 (+ 2223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:39.616023 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:39.616028 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:39.616033 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:39.616195 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:39.616527 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:39.616532 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:39.616534 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:39.617050 (+ 516us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:39.617256 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:39.617515 (+ 259us) server_negotiation.cc:299] Negotiation successful
0929 18:37:39.617878 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":239,"thread_start_us":120,"threads_started":1}
I20260929 18:37:39.619051 17859 raft_consensus.cc:686] T 00000000000000000000000000000000 P dc643846dbc84628ba4ec83ffbc08570 [term 1 LEADER]: Becoming Leader. State: Replica: dc643846dbc84628ba4ec83ffbc08570, State: Running, Role: LEADER
I20260929 18:37:39.619416 17859 consensus_queue.cc:237] T 00000000000000000000000000000000 P dc643846dbc84628ba4ec83ffbc08570 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc643846dbc84628ba4ec83ffbc08570" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 39601 } }
I20260929 18:37:39.619856 17854 sys_catalog.cc:564] T 00000000000000000000000000000000 P dc643846dbc84628ba4ec83ffbc08570 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:37:39.621618 17860 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc643846dbc84628ba4ec83ffbc08570 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dc643846dbc84628ba4ec83ffbc08570. Latest consensus state: current_term: 1 leader_uuid: "dc643846dbc84628ba4ec83ffbc08570" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc643846dbc84628ba4ec83ffbc08570" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 39601 } } }
I20260929 18:37:39.621753 17860 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc643846dbc84628ba4ec83ffbc08570 [sys.catalog]: This master's current role is: LEADER
I20260929 18:37:39.622097 17861 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc643846dbc84628ba4ec83ffbc08570 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dc643846dbc84628ba4ec83ffbc08570" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc643846dbc84628ba4ec83ffbc08570" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 39601 } } }
I20260929 18:37:39.622318 17861 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc643846dbc84628ba4ec83ffbc08570 [sys.catalog]: This master's current role is: LEADER
W20260929 18:37:39.624471 17873 catalog_manager.cc:1558] T 00000000000000000000000000000000 P dc643846dbc84628ba4ec83ffbc08570: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:37:39.624564 17873 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:37:39.624632 17874 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:37:39.625286 17874 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:37:39.630918 17874 catalog_manager.cc:1347] Generated new cluster ID: 4dd4e893b1e74084b2cf16d738afccec
I20260929 18:37:39.630977 17874 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:37:39.645449 17874 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:37:39.646811 17874 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:37:39.656562 17874 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dc643846dbc84628ba4ec83ffbc08570: Generated new TSK 0
I20260929 18:37:39.657358 17874 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.100.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.100.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.100.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.1.keytab.
Entry for principal kudu/127.13.100.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.1.keytab.
I20260929 18:37:39.693567 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707036041078-13712-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707036041078-13712-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707036041078-13712-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707036041078-13712-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.100.1:0
--local_ip_for_outbound_sockets=127.13.100.1
--webserver_interface=127.13.100.1
--webserver_port=0
--tserver_master_addrs=127.13.100.62:39601
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.1.keytab
--principal=kudu/127.13.100.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.100.20:39039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707036041078-13712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707036041078-13712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707036041078-13712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:37:39.804049 17880 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:37:39.804306 17880 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:37:39.804352 17880 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:37:39.808300 17880 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:37:39.808353 17880 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:37:39.808411 17880 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.100.1
I20260929 18:37:39.813657 17880 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.100.20:39039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707036041078-13712-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707036041078-13712-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.1.keytab
--principal=kudu/127.13.100.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.100.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707036041078-13712-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.100.1
--webserver_port=0
--tserver_master_addrs=127.13.100.62:39601
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17880
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.100.1
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707036041078-13712-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:37:39.814674 17880 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:37:39.815585 17880 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:37:39.824949 17888 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:37:39.824913 17885 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:37:39.824940 17886 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:37:39.825070 17880 server_base.cc:1034] running on GCE node
I20260929 18:37:39.825451 17880 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:37:39.826098 17880 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:37:39.827296 17880 hybrid_clock.cc:648] HybridClock initialized: now 1790707059827278 us; error 33 us; skew 500 ppm
Sep 29 18:37:39 dist-test-slave-j1xs krb5kdc[17774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707059, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:37:39.830008 17880 init.cc:377] Logged in from keytab as kudu/127.13.100.1@KRBTEST.COM (short username kudu)
I20260929 18:37:39.830713 17880 fs_manager.cc:359] Metadata directory not provided
I20260929 18:37:39.830807 17880 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:37:39.831007 17880 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:37:39.832974 17880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707036041078-13712-0/minicluster-data/ts-0/data/instance:
uuid: "1404be66936b491591040170516fcaf8"
format_stamp: "Formatted at 2026-09-29 18:37:39 on dist-test-slave-j1xs"
server_key: "baf52a447ceb393e6c5d5ed704e08b97"
server_key_iv: "87f721b9496dabe686f105a7c9224f9d"
server_key_version: "encryptionkey@0"
I20260929 18:37:39.833528 17880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707036041078-13712-0/minicluster-data/ts-0/wal/instance:
uuid: "1404be66936b491591040170516fcaf8"
format_stamp: "Formatted at 2026-09-29 18:37:39 on dist-test-slave-j1xs"
server_key: "baf52a447ceb393e6c5d5ed704e08b97"
server_key_iv: "87f721b9496dabe686f105a7c9224f9d"
server_key_version: "encryptionkey@0"
I20260929 18:37:39.837075 17880 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260929 18:37:39.839466 17894 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:37:39.840725 17880 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 18:37:39.840857 17880 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707036041078-13712-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707036041078-13712-0/minicluster-data/ts-0/wal
uuid: "1404be66936b491591040170516fcaf8"
format_stamp: "Formatted at 2026-09-29 18:37:39 on dist-test-slave-j1xs"
server_key: "baf52a447ceb393e6c5d5ed704e08b97"
server_key_iv: "87f721b9496dabe686f105a7c9224f9d"
server_key_version: "encryptionkey@0"
I20260929 18:37:39.840962 17880 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707036041078-13712-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707036041078-13712-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707036041078-13712-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:37:39.853772 17880 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:37:39.856904 17880 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:37:39.857120 17880 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:37:39.857697 17880 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:37:39.858572 17880 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:37:39.858644 17880 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:37:39.858711 17880 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:37:39.858759 17880 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:37:39.868162 17880 rpc_server.cc:307] RPC server started. Bound to: 127.13.100.1:34957
I20260929 18:37:39.868187 18007 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.100.1:34957 every 8 connection(s)
I20260929 18:37:39.868755 17880 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707036041078-13712-0/minicluster-data/ts-0/data/info.pb
I20260929 18:37:39.870319 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 17880
I20260929 18:37:39.870503 13712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707036041078-13712-0/minicluster-data/ts-0/wal/instance
I20260929 18:37:39.870853 13712 external_mini_cluster.cc:1428] Setting key 90df006e56c11314467774fd2ecaa1bd
WARNING: no policy specified for kudu/127.13.100.2@KRBTEST.COM; defaulting to no policy
Sep 29 18:37:39 dist-test-slave-j1xs krb5kdc[17774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707059, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.1@KRBTEST.COM for kudu/127.13.100.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.100.2@KRBTEST.COM" created.
I20260929 18:37:39.885529 17857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:39.871081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.1:49003 (local address 127.13.100.62:39601)
0929 18:37:39.871230 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:39.871235 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:39.872231 (+ 996us) server_negotiation.cc:407] Connection header received
0929 18:37:39.874280 (+ 2049us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:39.874286 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:39.874347 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:39.874430 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:39.875639 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:39.876567 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:39.877734 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:39.878205 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:39.878332 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:39.881513 (+ 3181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:39.881517 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:39.881519 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:37:39.881559 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:39.883058 (+ 1499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:39.883847 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:39.883851 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:39.883853 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:39.883952 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:39.884299 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:39.884305 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:39.884310 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:39.884612 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:39.884732 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:39.885240 (+ 508us) server_negotiation.cc:299] Negotiation successful
0929 18:37:39.885361 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 18:37:39.886564 18010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:39.871516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:39601 (local address 127.13.100.1:49003)
0929 18:37:39.872062 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:39.872133 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:39.874025 (+ 1892us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:39.874610 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:39.874617 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:39.874990 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:37:39.875479 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:39.875493 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:39.876730 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:39.876733 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:39.877600 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:39.877609 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:39.878522 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:39.878527 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:39.878792 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:39.879614 (+ 822us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:37:39.879635 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:37:39.881345 (+ 1710us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:37:39.883222 (+ 1877us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:39.883232 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:39.883257 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:39.883716 (+ 459us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:39.884060 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:39.884063 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:39.884064 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:39.884200 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:39.884734 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:37:39.884739 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:37:39.884994 (+ 255us) client_negotiation.cc:769] Sending connection context
0929 18:37:39.885199 (+ 205us) client_negotiation.cc:240] Negotiation successful
0929 18:37:39.885569 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":132,"threads_started":1}
I20260929 18:37:39.887986 18008 heartbeater.cc:347] Connected to a master server at 127.13.100.62:39601
I20260929 18:37:39.888255 18008 heartbeater.cc:464] Registering TS with master...
I20260929 18:37:39.888741 18008 heartbeater.cc:511] Master 127.13.100.62:39601 requested a full tablet report, sending...
I20260929 18:37:39.890372 17818 ts_manager.cc:194] Registered new tserver with Master: 1404be66936b491591040170516fcaf8 (127.13.100.1:34957)
I20260929 18:37:39.891525 17818 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.100.1@KRBTEST.COM'} at 127.13.100.1:49003
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.100.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.2.keytab.
Entry for principal kudu/127.13.100.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.2.keytab.
I20260929 18:37:39.901657 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707036041078-13712-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707036041078-13712-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707036041078-13712-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707036041078-13712-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.100.2:0
--local_ip_for_outbound_sockets=127.13.100.2
--webserver_interface=127.13.100.2
--webserver_port=0
--tserver_master_addrs=127.13.100.62:39601
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.2.keytab
--principal=kudu/127.13.100.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.100.20:39039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707036041078-13712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707036041078-13712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707036041078-13712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:37:40.013564 18013 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:37:40.013849 18013 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:37:40.013962 18013 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:37:40.018409 18013 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:37:40.018486 18013 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:37:40.018553 18013 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.100.2
I20260929 18:37:40.024645 18013 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.100.20:39039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707036041078-13712-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707036041078-13712-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.2.keytab
--principal=kudu/127.13.100.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.100.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707036041078-13712-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.100.2
--webserver_port=0
--tserver_master_addrs=127.13.100.62:39601
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18013
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.100.2
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707036041078-13712-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:37:40.025913 18013 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:37:40.026980 18013 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:37:40.036482 18018 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:37:40.036595 18013 server_base.cc:1034] running on GCE node
W20260929 18:37:40.036482 18021 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:37:40.036808 18019 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:37:40.037276 18013 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:37:40.038055 18013 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:37:40.039268 18013 hybrid_clock.cc:648] HybridClock initialized: now 1790707060039222 us; error 52 us; skew 500 ppm
Sep 29 18:37:40 dist-test-slave-j1xs krb5kdc[17774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707060, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:37:40.042409 18013 init.cc:377] Logged in from keytab as kudu/127.13.100.2@KRBTEST.COM (short username kudu)
I20260929 18:37:40.043236 18013 fs_manager.cc:359] Metadata directory not provided
I20260929 18:37:40.043334 18013 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:37:40.043571 18013 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:37:40.045789 18013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707036041078-13712-0/minicluster-data/ts-1/data/instance:
uuid: "613199c90419415e9689215ff66f3453"
format_stamp: "Formatted at 2026-09-29 18:37:40 on dist-test-slave-j1xs"
server_key: "15b2cedf60629a813255bee471d20db4"
server_key_iv: "deaa7b66bd0ae6419e56b26d02f4643f"
server_key_version: "encryptionkey@0"
I20260929 18:37:40.046469 18013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707036041078-13712-0/minicluster-data/ts-1/wal/instance:
uuid: "613199c90419415e9689215ff66f3453"
format_stamp: "Formatted at 2026-09-29 18:37:40 on dist-test-slave-j1xs"
server_key: "15b2cedf60629a813255bee471d20db4"
server_key_iv: "deaa7b66bd0ae6419e56b26d02f4643f"
server_key_version: "encryptionkey@0"
I20260929 18:37:40.050459 18013 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 18:37:40.053169 18027 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:37:40.054705 18013 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:37:40.054857 18013 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707036041078-13712-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707036041078-13712-0/minicluster-data/ts-1/wal
uuid: "613199c90419415e9689215ff66f3453"
format_stamp: "Formatted at 2026-09-29 18:37:40 on dist-test-slave-j1xs"
server_key: "15b2cedf60629a813255bee471d20db4"
server_key_iv: "deaa7b66bd0ae6419e56b26d02f4643f"
server_key_version: "encryptionkey@0"
I20260929 18:37:40.054979 18013 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707036041078-13712-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707036041078-13712-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707036041078-13712-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:37:40.097985 18013 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:37:40.101286 18013 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:37:40.101514 18013 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:37:40.102130 18013 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:37:40.102939 18013 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:37:40.103008 18013 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:37:40.103078 18013 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:37:40.103119 18013 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:37:40.112461 18013 rpc_server.cc:307] RPC server started. Bound to: 127.13.100.2:37307
I20260929 18:37:40.112478 18140 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.100.2:37307 every 8 connection(s)
I20260929 18:37:40.113073 18013 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707036041078-13712-0/minicluster-data/ts-1/data/info.pb
I20260929 18:37:40.118783 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 18013
I20260929 18:37:40.118878 13712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707036041078-13712-0/minicluster-data/ts-1/wal/instance
I20260929 18:37:40.119164 13712 external_mini_cluster.cc:1428] Setting key 3f98e4f54a48b0ab187f94ce5bf8279e
Sep 29 18:37:40 dist-test-slave-j1xs krb5kdc[17774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707060, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.2@KRBTEST.COM for kudu/127.13.100.62@KRBTEST.COM
I20260929 18:37:40.127566 17857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:40.114821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.2:41525 (local address 127.13.100.62:39601)
0929 18:37:40.114964 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:40.114968 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:40.115721 (+ 753us) server_negotiation.cc:407] Connection header received
0929 18:37:40.116950 (+ 1229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:40.116957 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:40.117002 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:40.117082 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:40.118424 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:40.119083 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:40.120556 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:40.120905 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:40.121023 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:40.123895 (+ 2872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:40.123898 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:40.123900 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:37:40.123922 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:40.125437 (+ 1515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:40.126047 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:40.126050 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:40.126051 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:40.126128 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:40.126475 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:40.126478 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:40.126479 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:40.126664 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:40.126772 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:40.127279 (+ 507us) server_negotiation.cc:299] Negotiation successful
0929 18:37:40.127405 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 18:37:40.128386 18143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:40.115089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:39601 (local address 127.13.100.2:41525)
0929 18:37:40.115546 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:40.115587 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:40.116745 (+ 1158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:40.117241 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:40.117248 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:40.117590 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:37:40.118271 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:40.118287 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:40.119213 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:40.119216 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:40.120445 (+ 1229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:40.120455 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:40.121095 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:40.121099 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:40.121362 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:40.122132 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:37:40.122154 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:37:40.123710 (+ 1556us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:37:40.125573 (+ 1863us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:40.125579 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:40.125594 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:40.125855 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:40.126225 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:40.126229 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:40.126231 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:40.126371 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:40.126771 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:37:40.126778 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:37:40.127031 (+ 253us) client_negotiation.cc:769] Sending connection context
0929 18:37:40.127237 (+ 206us) client_negotiation.cc:240] Negotiation successful
0929 18:37:40.127571 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":115,"threads_started":1}
WARNING: no policy specified for kudu/127.13.100.3@KRBTEST.COM; defaulting to no policy
I20260929 18:37:40.129565 18141 heartbeater.cc:347] Connected to a master server at 127.13.100.62:39601
I20260929 18:37:40.129818 18141 heartbeater.cc:464] Registering TS with master...
I20260929 18:37:40.130367 18141 heartbeater.cc:511] Master 127.13.100.62:39601 requested a full tablet report, sending...
I20260929 18:37:40.131419 17818 ts_manager.cc:194] Registered new tserver with Master: 613199c90419415e9689215ff66f3453 (127.13.100.2:37307)
I20260929 18:37:40.132280 17818 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.100.2@KRBTEST.COM'} at 127.13.100.2:41525
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.100.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.100.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.3.keytab.
Entry for principal kudu/127.13.100.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.3.keytab.
I20260929 18:37:40.148782 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707036041078-13712-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707036041078-13712-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707036041078-13712-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707036041078-13712-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.100.3:0
--local_ip_for_outbound_sockets=127.13.100.3
--webserver_interface=127.13.100.3
--webserver_port=0
--tserver_master_addrs=127.13.100.62:39601
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.3.keytab
--principal=kudu/127.13.100.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.100.20:39039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707036041078-13712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707036041078-13712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707036041078-13712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:37:40.258761 18146 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:37:40.259104 18146 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:37:40.259203 18146 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:37:40.263203 18146 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:37:40.263262 18146 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:37:40.263324 18146 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.100.3
I20260929 18:37:40.268548 18146 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.100.20:39039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707036041078-13712-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707036041078-13712-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.3.keytab
--principal=kudu/127.13.100.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.100.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707036041078-13712-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.100.3
--webserver_port=0
--tserver_master_addrs=127.13.100.62:39601
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18146
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.100.3
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707036041078-13712-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:37:40.269625 18146 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:37:40.270642 18146 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:37:40.280756 18152 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:37:40.280803 18151 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:37:40.280756 18154 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:37:40.281024 18146 server_base.cc:1034] running on GCE node
I20260929 18:37:40.281616 18146 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:37:40.282357 18146 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:37:40.283576 18146 hybrid_clock.cc:648] HybridClock initialized: now 1790707060283556 us; error 33 us; skew 500 ppm
Sep 29 18:37:40 dist-test-slave-j1xs krb5kdc[17774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707060, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:37:40.287132 18146 init.cc:377] Logged in from keytab as kudu/127.13.100.3@KRBTEST.COM (short username kudu)
I20260929 18:37:40.288266 18146 fs_manager.cc:359] Metadata directory not provided
I20260929 18:37:40.288383 18146 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:37:40.288636 18146 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:37:40.290817 18146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707036041078-13712-0/minicluster-data/ts-2/data/instance:
uuid: "15d7cf0a00554349bec73bc7a957a31e"
format_stamp: "Formatted at 2026-09-29 18:37:40 on dist-test-slave-j1xs"
server_key: "4132e5a50f714422f7a574b5ff5df6de"
server_key_iv: "7d550ea3c3e5dc87860d93ee81486c36"
server_key_version: "encryptionkey@0"
I20260929 18:37:40.291407 18146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707036041078-13712-0/minicluster-data/ts-2/wal/instance:
uuid: "15d7cf0a00554349bec73bc7a957a31e"
format_stamp: "Formatted at 2026-09-29 18:37:40 on dist-test-slave-j1xs"
server_key: "4132e5a50f714422f7a574b5ff5df6de"
server_key_iv: "7d550ea3c3e5dc87860d93ee81486c36"
server_key_version: "encryptionkey@0"
I20260929 18:37:40.295238 18146 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 18:37:40.297744 18160 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:37:40.299207 18146 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 18:37:40.299356 18146 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707036041078-13712-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707036041078-13712-0/minicluster-data/ts-2/wal
uuid: "15d7cf0a00554349bec73bc7a957a31e"
format_stamp: "Formatted at 2026-09-29 18:37:40 on dist-test-slave-j1xs"
server_key: "4132e5a50f714422f7a574b5ff5df6de"
server_key_iv: "7d550ea3c3e5dc87860d93ee81486c36"
server_key_version: "encryptionkey@0"
I20260929 18:37:40.299474 18146 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707036041078-13712-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707036041078-13712-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707036041078-13712-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:37:40.317129 18146 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:37:40.320429 18146 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:37:40.320648 18146 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:37:40.321342 18146 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:37:40.322242 18146 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:37:40.322319 18146 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:37:40.322391 18146 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:37:40.322444 18146 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:37:40.331900 18146 rpc_server.cc:307] RPC server started. Bound to: 127.13.100.3:43687
I20260929 18:37:40.331926 18273 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.100.3:43687 every 8 connection(s)
I20260929 18:37:40.332579 18146 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707036041078-13712-0/minicluster-data/ts-2/data/info.pb
I20260929 18:37:40.335108 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 18146
I20260929 18:37:40.335250 13712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707036041078-13712-0/minicluster-data/ts-2/wal/instance
I20260929 18:37:40.335649 13712 external_mini_cluster.cc:1428] Setting key 6b18cf8f255b6e08dd8f5e9fd577dcf4
Sep 29 18:37:40 dist-test-slave-j1xs krb5kdc[17774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707060, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.3@KRBTEST.COM for kudu/127.13.100.62@KRBTEST.COM
I20260929 18:37:40.347317 17857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:40.334506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.3:58851 (local address 127.13.100.62:39601)
0929 18:37:40.334730 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:40.334737 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:40.335555 (+ 818us) server_negotiation.cc:407] Connection header received
0929 18:37:40.337552 (+ 1997us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:40.337555 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:40.337614 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:40.337693 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:40.339004 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:40.339534 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:40.340665 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:40.340894 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:40.341004 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:40.343707 (+ 2703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:40.343710 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:40.343712 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:37:40.343738 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:40.345281 (+ 1543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:40.345833 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:40.345836 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:40.345837 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:40.345962 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:40.346315 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:40.346318 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:40.346327 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:40.346503 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:40.346584 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:40.347097 (+ 513us) server_negotiation.cc:299] Negotiation successful
0929 18:37:40.347196 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260929 18:37:40.348153 18276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:40.334802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:39601 (local address 127.13.100.3:58851)
0929 18:37:40.335356 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:40.335420 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:40.337345 (+ 1925us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:40.337879 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:40.337926 (+ 47us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:40.338381 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:37:40.338861 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:40.338874 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:40.339664 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:40.339667 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:40.340535 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:40.340543 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:40.341039 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:40.341043 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:40.341222 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:40.341946 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:37:40.341967 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:37:40.343556 (+ 1589us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:37:40.345393 (+ 1837us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:40.345398 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:40.345410 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:40.345738 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:40.346081 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:40.346083 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:40.346085 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:40.346231 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:40.346628 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:37:40.346635 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:37:40.346872 (+ 237us) client_negotiation.cc:769] Sending connection context
0929 18:37:40.347043 (+ 171us) client_negotiation.cc:240] Negotiation successful
0929 18:37:40.347306 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":104,"threads_started":1}
I20260929 18:37:40.349345 18274 heartbeater.cc:347] Connected to a master server at 127.13.100.62:39601
I20260929 18:37:40.349591 18274 heartbeater.cc:464] Registering TS with master...
I20260929 18:37:40.350111 18274 heartbeater.cc:511] Master 127.13.100.62:39601 requested a full tablet report, sending...
I20260929 18:37:40.351068 17817 ts_manager.cc:194] Registered new tserver with Master: 15d7cf0a00554349bec73bc7a957a31e (127.13.100.3:43687)
I20260929 18:37:40.351686 17817 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.100.3@KRBTEST.COM'} at 127.13.100.3:58851
I20260929 18:37:40.361999 13712 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 18:37:40.371874 17857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:40.364046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42416 (local address 127.13.100.62:39601)
0929 18:37:40.364196 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:40.364200 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:40.364355 (+ 155us) server_negotiation.cc:407] Connection header received
0929 18:37:40.364509 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:40.364513 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:40.364572 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:40.364658 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:40.365373 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:40.365938 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:40.366941 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:40.367175 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:40.367286 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:40.368626 (+ 1340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:40.368630 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:40.368633 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:37:40.368659 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:40.370155 (+ 1496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:40.370573 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:40.370578 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:40.370580 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:40.370668 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:40.370936 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:40.370941 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:40.370942 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:40.371166 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:40.371304 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:40.371542 (+ 238us) server_negotiation.cc:299] Negotiation successful
0929 18:37:40.371661 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 18:37:40.375154 17817 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42416:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:37:40.378084 17817 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:37:40.392998 18289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:40.387392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.1:34957 (local address 127.0.0.1:47744)
0929 18:37:40.387713 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:40.387725 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:40.387797 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:40.388912 (+ 1115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:40.388915 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:40.388932 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:37:40.389024 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:40.389033 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:40.390251 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:40.390254 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:40.391631 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:40.391642 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:40.392613 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:40.392616 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:40.392710 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:40.392730 (+ 20us) client_negotiation.cc:769] Sending connection context
0929 18:37:40.392803 (+ 73us) client_negotiation.cc:240] Negotiation successful
0929 18:37:40.392854 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"mutex_wait_us":91,"thread_start_us":119,"threads_started":1}
I20260929 18:37:40.393066 18290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:40.387996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47744 (local address 127.13.100.1:34957)
0929 18:37:40.388372 (+ 376us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:40.388380 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:40.388400 (+ 20us) server_negotiation.cc:407] Connection header received
0929 18:37:40.388471 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:40.388483 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:40.388772 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:40.388933 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:37:40.389159 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:40.390142 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:40.391766 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:40.392492 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:40.392608 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:40.392737 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:40.392833 (+ 96us) server_negotiation.cc:299] Negotiation successful
0929 18:37:40.392924 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":83,"threads_started":1}
I20260929 18:37:40.393541 18287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:40.386636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57112 (local address 127.13.100.3:43687)
0929 18:37:40.387448 (+ 812us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:40.387466 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:40.387498 (+ 32us) server_negotiation.cc:407] Connection header received
0929 18:37:40.387670 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:40.387677 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:40.387900 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:40.388046 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:37:40.388616 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:40.389915 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:40.391499 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:40.392277 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:40.392393 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:40.392544 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:40.393062 (+ 518us) server_negotiation.cc:299] Negotiation successful
0929 18:37:40.393170 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":495,"thread_start_us":164,"threads_started":1}
I20260929 18:37:40.393945 18286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:40.386610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.2:37307 (local address 127.0.0.1:41676)
0929 18:37:40.387405 (+ 795us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:40.387416 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:40.387480 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:40.388215 (+ 735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:40.388221 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:40.388247 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:37:40.388469 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:40.388477 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:40.390166 (+ 1689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:40.390171 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:40.391632 (+ 1461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:40.391642 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:40.393516 (+ 1874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:40.393520 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:40.393623 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:40.393644 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 18:37:40.393709 (+ 65us) client_negotiation.cc:240] Negotiation successful
0929 18:37:40.393776 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":725,"mutex_wait_us":140,"spinlock_wait_cycles":8192,"thread_start_us":93,"threads_started":1}
I20260929 18:37:40.394253 18288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:40.386795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41676 (local address 127.13.100.2:37307)
0929 18:37:40.387689 (+ 894us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:40.387702 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:40.387725 (+ 23us) server_negotiation.cc:407] Connection header received
0929 18:37:40.387807 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:40.387812 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:40.388021 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:40.388659 (+ 638us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:37:40.388841 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:40.389981 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:40.392618 (+ 2637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:40.393388 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:40.393515 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:40.393924 (+ 409us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:40.393977 (+ 53us) server_negotiation.cc:299] Negotiation successful
0929 18:37:40.394074 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":793,"thread_start_us":109,"threads_started":1}
I20260929 18:37:40.392998 18285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:40.386518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.3:43687 (local address 127.0.0.1:57112)
0929 18:37:40.387305 (+ 787us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:40.387357 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:40.387480 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:40.388255 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:40.388258 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:40.388286 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:37:40.388471 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:40.388478 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:40.390053 (+ 1575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:40.390056 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:40.391369 (+ 1313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:40.391378 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:40.392413 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:40.392416 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:40.392508 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:40.392570 (+ 62us) client_negotiation.cc:769] Sending connection context
0929 18:37:40.392694 (+ 124us) client_negotiation.cc:240] Negotiation successful
0929 18:37:40.392825 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":615,"mutex_wait_us":66,"spinlock_wait_cycles":768,"thread_start_us":85,"threads_started":1}
I20260929 18:37:40.397172 18208 tablet_service.cc:1467] Processing CreateTablet for tablet dd188857a8a744ea86ff35faefe8a310 (DEFAULT_TABLE table=test-table [id=c34f0f963c174fc1ae40f999428534b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:37:40.397207 17942 tablet_service.cc:1467] Processing CreateTablet for tablet dd188857a8a744ea86ff35faefe8a310 (DEFAULT_TABLE table=test-table [id=c34f0f963c174fc1ae40f999428534b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:37:40.397205 18075 tablet_service.cc:1467] Processing CreateTablet for tablet dd188857a8a744ea86ff35faefe8a310 (DEFAULT_TABLE table=test-table [id=c34f0f963c174fc1ae40f999428534b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:37:40.398283 17942 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd188857a8a744ea86ff35faefe8a310. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:37:40.398284 18075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd188857a8a744ea86ff35faefe8a310. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:37:40.398298 18208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd188857a8a744ea86ff35faefe8a310. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:37:40.404584 18291 tablet_bootstrap.cc:492] T dd188857a8a744ea86ff35faefe8a310 P 15d7cf0a00554349bec73bc7a957a31e: Bootstrap starting.
I20260929 18:37:40.406154 18293 tablet_bootstrap.cc:492] T dd188857a8a744ea86ff35faefe8a310 P 613199c90419415e9689215ff66f3453: Bootstrap starting.
I20260929 18:37:40.406154 18292 tablet_bootstrap.cc:492] T dd188857a8a744ea86ff35faefe8a310 P 1404be66936b491591040170516fcaf8: Bootstrap starting.
I20260929 18:37:40.406857 18291 tablet_bootstrap.cc:654] T dd188857a8a744ea86ff35faefe8a310 P 15d7cf0a00554349bec73bc7a957a31e: Neither blocks nor log segments found. Creating new log.
I20260929 18:37:40.407938 18291 log.cc:824] T dd188857a8a744ea86ff35faefe8a310 P 15d7cf0a00554349bec73bc7a957a31e: Log is configured to *not* fsync() on all Append() calls
I20260929 18:37:40.408468 18292 tablet_bootstrap.cc:654] T dd188857a8a744ea86ff35faefe8a310 P 1404be66936b491591040170516fcaf8: Neither blocks nor log segments found. Creating new log.
I20260929 18:37:40.408471 18293 tablet_bootstrap.cc:654] T dd188857a8a744ea86ff35faefe8a310 P 613199c90419415e9689215ff66f3453: Neither blocks nor log segments found. Creating new log.
I20260929 18:37:40.409438 18292 log.cc:824] T dd188857a8a744ea86ff35faefe8a310 P 1404be66936b491591040170516fcaf8: Log is configured to *not* fsync() on all Append() calls
I20260929 18:37:40.409698 18293 log.cc:824] T dd188857a8a744ea86ff35faefe8a310 P 613199c90419415e9689215ff66f3453: Log is configured to *not* fsync() on all Append() calls
I20260929 18:37:40.410173 18291 tablet_bootstrap.cc:492] T dd188857a8a744ea86ff35faefe8a310 P 15d7cf0a00554349bec73bc7a957a31e: No bootstrap required, opened a new log
I20260929 18:37:40.410378 18291 ts_tablet_manager.cc:1397] T dd188857a8a744ea86ff35faefe8a310 P 15d7cf0a00554349bec73bc7a957a31e: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260929 18:37:40.411538 18293 tablet_bootstrap.cc:492] T dd188857a8a744ea86ff35faefe8a310 P 613199c90419415e9689215ff66f3453: No bootstrap required, opened a new log
I20260929 18:37:40.411538 18292 tablet_bootstrap.cc:492] T dd188857a8a744ea86ff35faefe8a310 P 1404be66936b491591040170516fcaf8: No bootstrap required, opened a new log
I20260929 18:37:40.411731 18293 ts_tablet_manager.cc:1397] T dd188857a8a744ea86ff35faefe8a310 P 613199c90419415e9689215ff66f3453: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260929 18:37:40.411731 18292 ts_tablet_manager.cc:1397] T dd188857a8a744ea86ff35faefe8a310 P 1404be66936b491591040170516fcaf8: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 18:37:40.413373 18291 raft_consensus.cc:348] T dd188857a8a744ea86ff35faefe8a310 P 15d7cf0a00554349bec73bc7a957a31e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d7cf0a00554349bec73bc7a957a31e" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 43687 } } peers { permanent_uuid: "613199c90419415e9689215ff66f3453" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 37307 } } peers { permanent_uuid: "1404be66936b491591040170516fcaf8" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 34957 } }
I20260929 18:37:40.413633 18291 raft_consensus.cc:374] T dd188857a8a744ea86ff35faefe8a310 P 15d7cf0a00554349bec73bc7a957a31e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:37:40.413687 18291 raft_consensus.cc:729] T dd188857a8a744ea86ff35faefe8a310 P 15d7cf0a00554349bec73bc7a957a31e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15d7cf0a00554349bec73bc7a957a31e, State: Initialized, Role: FOLLOWER
I20260929 18:37:40.414173 18291 consensus_queue.cc:260] T dd188857a8a744ea86ff35faefe8a310 P 15d7cf0a00554349bec73bc7a957a31e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d7cf0a00554349bec73bc7a957a31e" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 43687 } } peers { permanent_uuid: "613199c90419415e9689215ff66f3453" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 37307 } } peers { permanent_uuid: "1404be66936b491591040170516fcaf8" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 34957 } }
I20260929 18:37:40.414865 18274 heartbeater.cc:503] Master 127.13.100.62:39601 was elected leader, sending a full tablet report...
I20260929 18:37:40.415179 18291 ts_tablet_manager.cc:1428] T dd188857a8a744ea86ff35faefe8a310 P 15d7cf0a00554349bec73bc7a957a31e: Time spent starting tablet: real 0.005s user 0.001s sys 0.003s
I20260929 18:37:40.415853 18293 raft_consensus.cc:348] T dd188857a8a744ea86ff35faefe8a310 P 613199c90419415e9689215ff66f3453 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d7cf0a00554349bec73bc7a957a31e" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 43687 } } peers { permanent_uuid: "613199c90419415e9689215ff66f3453" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 37307 } } peers { permanent_uuid: "1404be66936b491591040170516fcaf8" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 34957 } }
I20260929 18:37:40.416136 18293 raft_consensus.cc:374] T dd188857a8a744ea86ff35faefe8a310 P 613199c90419415e9689215ff66f3453 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:37:40.416201 18293 raft_consensus.cc:729] T dd188857a8a744ea86ff35faefe8a310 P 613199c90419415e9689215ff66f3453 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 613199c90419415e9689215ff66f3453, State: Initialized, Role: FOLLOWER
I20260929 18:37:40.416668 18293 consensus_queue.cc:260] T dd188857a8a744ea86ff35faefe8a310 P 613199c90419415e9689215ff66f3453 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d7cf0a00554349bec73bc7a957a31e" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 43687 } } peers { permanent_uuid: "613199c90419415e9689215ff66f3453" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 37307 } } peers { permanent_uuid: "1404be66936b491591040170516fcaf8" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 34957 } }
I20260929 18:37:40.416865 18292 raft_consensus.cc:348] T dd188857a8a744ea86ff35faefe8a310 P 1404be66936b491591040170516fcaf8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d7cf0a00554349bec73bc7a957a31e" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 43687 } } peers { permanent_uuid: "613199c90419415e9689215ff66f3453" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 37307 } } peers { permanent_uuid: "1404be66936b491591040170516fcaf8" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 34957 } }
I20260929 18:37:40.417140 18292 raft_consensus.cc:374] T dd188857a8a744ea86ff35faefe8a310 P 1404be66936b491591040170516fcaf8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:37:40.417204 18292 raft_consensus.cc:729] T dd188857a8a744ea86ff35faefe8a310 P 1404be66936b491591040170516fcaf8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1404be66936b491591040170516fcaf8, State: Initialized, Role: FOLLOWER
I20260929 18:37:40.417506 18141 heartbeater.cc:503] Master 127.13.100.62:39601 was elected leader, sending a full tablet report...
I20260929 18:37:40.417747 18293 ts_tablet_manager.cc:1428] T dd188857a8a744ea86ff35faefe8a310 P 613199c90419415e9689215ff66f3453: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260929 18:37:40.417652 18292 consensus_queue.cc:260] T dd188857a8a744ea86ff35faefe8a310 P 1404be66936b491591040170516fcaf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d7cf0a00554349bec73bc7a957a31e" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 43687 } } peers { permanent_uuid: "613199c90419415e9689215ff66f3453" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 37307 } } peers { permanent_uuid: "1404be66936b491591040170516fcaf8" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 34957 } }
I20260929 18:37:40.418327 18008 heartbeater.cc:503] Master 127.13.100.62:39601 was elected leader, sending a full tablet report...
I20260929 18:37:40.418681 18292 ts_tablet_manager.cc:1428] T dd188857a8a744ea86ff35faefe8a310 P 1404be66936b491591040170516fcaf8: Time spent starting tablet: real 0.007s user 0.007s sys 0.001s
I20260929 18:37:40.554253 18299 raft_consensus.cc:482] T dd188857a8a744ea86ff35faefe8a310 P 1404be66936b491591040170516fcaf8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:37:40.554447 18299 raft_consensus.cc:504] T dd188857a8a744ea86ff35faefe8a310 P 1404be66936b491591040170516fcaf8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d7cf0a00554349bec73bc7a957a31e" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 43687 } } peers { permanent_uuid: "613199c90419415e9689215ff66f3453" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 37307 } } peers { permanent_uuid: "1404be66936b491591040170516fcaf8" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 34957 } }
I20260929 18:37:40.555550 18299 leader_election.cc:288] T dd188857a8a744ea86ff35faefe8a310 P 1404be66936b491591040170516fcaf8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 15d7cf0a00554349bec73bc7a957a31e (127.13.100.3:43687), 613199c90419415e9689215ff66f3453 (127.13.100.2:37307)
I20260929 18:37:40.559608 18288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:40.556205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.1:45833 (local address 127.13.100.2:37307)
0929 18:37:40.556351 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:40.556355 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:40.556371 (+ 16us) server_negotiation.cc:407] Connection header received
0929 18:37:40.556454 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:40.556457 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:40.556502 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:40.556579 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:37:40.556854 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:40.557353 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:40.558518 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:40.558981 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:40.559120 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:40.559172 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:40.559396 (+ 224us) server_negotiation.cc:299] Negotiation successful
0929 18:37:40.559467 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 18:37:40.559639 18287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:40.555886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.1:42815 (local address 127.13.100.3:43687)
0929 18:37:40.556035 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:40.556041 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:40.556144 (+ 103us) server_negotiation.cc:407] Connection header received
0929 18:37:40.556373 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:40.556377 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:40.556427 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:40.556511 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:37:40.556805 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:40.557299 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:40.558531 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:40.558995 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:40.559129 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:40.559185 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:40.559428 (+ 243us) server_negotiation.cc:299] Negotiation successful
0929 18:37:40.559508 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 18:37:40.559676 18300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:40.555762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.3:43687 (local address 127.13.100.1:42815)
0929 18:37:40.556051 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:40.556066 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:40.556240 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:40.556524 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:40.556527 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:40.556543 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:37:40.556697 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:40.556703 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:40.557410 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:40.557415 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:40.558418 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:40.558427 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:40.559130 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:40.559133 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:40.559293 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:40.559310 (+ 17us) client_negotiation.cc:769] Sending connection context
0929 18:37:40.559398 (+ 88us) client_negotiation.cc:240] Negotiation successful
0929 18:37:40.559530 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"mutex_wait_us":112,"thread_start_us":119,"threads_started":1}
I20260929 18:37:40.559846 18301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:40.555882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.2:37307 (local address 127.13.100.1:45833)
0929 18:37:40.556221 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:40.556254 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:40.556363 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:40.556598 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:40.556600 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:40.556616 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:37:40.556742 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:40.556748 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:40.557463 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:40.557469 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:40.558398 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:40.558407 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:40.559120 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:40.559124 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:40.559251 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:40.559292 (+ 41us) client_negotiation.cc:769] Sending connection context
0929 18:37:40.559381 (+ 89us) client_negotiation.cc:240] Negotiation successful
0929 18:37:40.559479 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"mutex_wait_us":62,"thread_start_us":122,"threads_started":1}
I20260929 18:37:40.560340 18228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd188857a8a744ea86ff35faefe8a310" candidate_uuid: "1404be66936b491591040170516fcaf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15d7cf0a00554349bec73bc7a957a31e" is_pre_election: true
I20260929 18:37:40.560454 18095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd188857a8a744ea86ff35faefe8a310" candidate_uuid: "1404be66936b491591040170516fcaf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "613199c90419415e9689215ff66f3453" is_pre_election: true
I20260929 18:37:40.560684 18228 raft_consensus.cc:2454] T dd188857a8a744ea86ff35faefe8a310 P 15d7cf0a00554349bec73bc7a957a31e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1404be66936b491591040170516fcaf8 in term 0.
I20260929 18:37:40.560779 18095 raft_consensus.cc:2454] T dd188857a8a744ea86ff35faefe8a310 P 613199c90419415e9689215ff66f3453 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1404be66936b491591040170516fcaf8 in term 0.
I20260929 18:37:40.561161 17895 leader_election.cc:302] T dd188857a8a744ea86ff35faefe8a310 P 1404be66936b491591040170516fcaf8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1404be66936b491591040170516fcaf8, 15d7cf0a00554349bec73bc7a957a31e; no voters:
I20260929 18:37:40.561424 18299 raft_consensus.cc:2781] T dd188857a8a744ea86ff35faefe8a310 P 1404be66936b491591040170516fcaf8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:37:40.561517 18299 raft_consensus.cc:482] T dd188857a8a744ea86ff35faefe8a310 P 1404be66936b491591040170516fcaf8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:37:40.561578 18299 raft_consensus.cc:3037] T dd188857a8a744ea86ff35faefe8a310 P 1404be66936b491591040170516fcaf8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:37:40.562793 18299 raft_consensus.cc:504] T dd188857a8a744ea86ff35faefe8a310 P 1404be66936b491591040170516fcaf8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d7cf0a00554349bec73bc7a957a31e" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 43687 } } peers { permanent_uuid: "613199c90419415e9689215ff66f3453" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 37307 } } peers { permanent_uuid: "1404be66936b491591040170516fcaf8" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 34957 } }
I20260929 18:37:40.563133 18299 leader_election.cc:288] T dd188857a8a744ea86ff35faefe8a310 P 1404be66936b491591040170516fcaf8 [CANDIDATE]: Term 1 election: Requested vote from peers 15d7cf0a00554349bec73bc7a957a31e (127.13.100.3:43687), 613199c90419415e9689215ff66f3453 (127.13.100.2:37307)
I20260929 18:37:40.563473 18228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd188857a8a744ea86ff35faefe8a310" candidate_uuid: "1404be66936b491591040170516fcaf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15d7cf0a00554349bec73bc7a957a31e"
I20260929 18:37:40.563473 18095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd188857a8a744ea86ff35faefe8a310" candidate_uuid: "1404be66936b491591040170516fcaf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "613199c90419415e9689215ff66f3453"
I20260929 18:37:40.563588 18228 raft_consensus.cc:3037] T dd188857a8a744ea86ff35faefe8a310 P 15d7cf0a00554349bec73bc7a957a31e [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:37:40.563611 18095 raft_consensus.cc:3037] T dd188857a8a744ea86ff35faefe8a310 P 613199c90419415e9689215ff66f3453 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:37:40.564862 18228 raft_consensus.cc:2454] T dd188857a8a744ea86ff35faefe8a310 P 15d7cf0a00554349bec73bc7a957a31e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1404be66936b491591040170516fcaf8 in term 1.
I20260929 18:37:40.564862 18095 raft_consensus.cc:2454] T dd188857a8a744ea86ff35faefe8a310 P 613199c90419415e9689215ff66f3453 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1404be66936b491591040170516fcaf8 in term 1.
I20260929 18:37:40.565237 17896 leader_election.cc:302] T dd188857a8a744ea86ff35faefe8a310 P 1404be66936b491591040170516fcaf8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1404be66936b491591040170516fcaf8, 15d7cf0a00554349bec73bc7a957a31e, 613199c90419415e9689215ff66f3453; no voters:
I20260929 18:37:40.565442 18299 raft_consensus.cc:2781] T dd188857a8a744ea86ff35faefe8a310 P 1404be66936b491591040170516fcaf8 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:37:40.565672 18299 raft_consensus.cc:686] T dd188857a8a744ea86ff35faefe8a310 P 1404be66936b491591040170516fcaf8 [term 1 LEADER]: Becoming Leader. State: Replica: 1404be66936b491591040170516fcaf8, State: Running, Role: LEADER
I20260929 18:37:40.566035 18299 consensus_queue.cc:237] T dd188857a8a744ea86ff35faefe8a310 P 1404be66936b491591040170516fcaf8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d7cf0a00554349bec73bc7a957a31e" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 43687 } } peers { permanent_uuid: "613199c90419415e9689215ff66f3453" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 37307 } } peers { permanent_uuid: "1404be66936b491591040170516fcaf8" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 34957 } }
I20260929 18:37:40.569046 17817 catalog_manager.cc:5495] T dd188857a8a744ea86ff35faefe8a310 P 1404be66936b491591040170516fcaf8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1404be66936b491591040170516fcaf8 (127.13.100.1). New cstate: current_term: 1 leader_uuid: "1404be66936b491591040170516fcaf8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d7cf0a00554349bec73bc7a957a31e" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 43687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "613199c90419415e9689215ff66f3453" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 37307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1404be66936b491591040170516fcaf8" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 34957 } health_report { overall_health: HEALTHY } } }
I20260929 18:37:40.581301 18290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:40.577758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47754 (local address 127.13.100.1:34957)
0929 18:37:40.577878 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:40.577905 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:40.577931 (+ 26us) server_negotiation.cc:407] Connection header received
0929 18:37:40.578132 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:40.578136 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:40.578186 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:40.578295 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:37:40.578541 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:40.579045 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:40.579915 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:40.580143 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:40.580253 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:40.580548 (+ 295us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:37:40.580930 (+ 382us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:37:40.581060 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:40.581103 (+ 43us) server_negotiation.cc:299] Negotiation successful
0929 18:37:40.581165 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
W20260929 18:37:40.583421 18275 tablet.cc:2369] T dd188857a8a744ea86ff35faefe8a310 P 15d7cf0a00554349bec73bc7a957a31e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:37:40.587093 18228 raft_consensus.cc:1261] T dd188857a8a744ea86ff35faefe8a310 P 15d7cf0a00554349bec73bc7a957a31e [term 1 FOLLOWER]: Refusing update from remote peer 1404be66936b491591040170516fcaf8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:37:40.587095 18095 raft_consensus.cc:1261] T dd188857a8a744ea86ff35faefe8a310 P 613199c90419415e9689215ff66f3453 [term 1 FOLLOWER]: Refusing update from remote peer 1404be66936b491591040170516fcaf8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:37:40.587883 18302 consensus_queue.cc:1035] T dd188857a8a744ea86ff35faefe8a310 P 1404be66936b491591040170516fcaf8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15d7cf0a00554349bec73bc7a957a31e" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 43687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:37:40.588135 18299 consensus_queue.cc:1035] T dd188857a8a744ea86ff35faefe8a310 P 1404be66936b491591040170516fcaf8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "613199c90419415e9689215ff66f3453" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 37307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:37:40.604228 17817 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42416:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:37:40.604411 17817 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42416:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:37:40.606917 17817 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dc643846dbc84628ba4ec83ffbc08570: Sending DeleteTablet for 3 replicas of tablet dd188857a8a744ea86ff35faefe8a310
I20260929 18:37:40.607606 18208 tablet_service.cc:1514] Processing DeleteTablet for tablet dd188857a8a744ea86ff35faefe8a310 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:37:40 UTC) from {username='kudu', principal='kudu/127.13.100.62@KRBTEST.COM'} at 127.0.0.1:57112
I20260929 18:37:40.607793 18075 tablet_service.cc:1514] Processing DeleteTablet for tablet dd188857a8a744ea86ff35faefe8a310 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:37:40 UTC) from {username='kudu', principal='kudu/127.13.100.62@KRBTEST.COM'} at 127.0.0.1:41676
I20260929 18:37:40.608148 18314 tablet_replica.cc:330] T dd188857a8a744ea86ff35faefe8a310 P 15d7cf0a00554349bec73bc7a957a31e: stopping tablet replica
I20260929 18:37:40.608456 18314 raft_consensus.cc:2229] T dd188857a8a744ea86ff35faefe8a310 P 15d7cf0a00554349bec73bc7a957a31e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:37:40.608517 17942 tablet_service.cc:1514] Processing DeleteTablet for tablet dd188857a8a744ea86ff35faefe8a310 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:37:40 UTC) from {username='kudu', principal='kudu/127.13.100.62@KRBTEST.COM'} at 127.0.0.1:47744
I20260929 18:37:40.608860 18315 tablet_replica.cc:330] T dd188857a8a744ea86ff35faefe8a310 P 613199c90419415e9689215ff66f3453: stopping tablet replica
I20260929 18:37:40.609122 18315 raft_consensus.cc:2229] T dd188857a8a744ea86ff35faefe8a310 P 613199c90419415e9689215ff66f3453 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:37:40.609211 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 17880
I20260929 18:37:40.610074 18315 raft_consensus.cc:2258] T dd188857a8a744ea86ff35faefe8a310 P 613199c90419415e9689215ff66f3453 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:37:40.610636 18314 raft_consensus.cc:2258] T dd188857a8a744ea86ff35faefe8a310 P 15d7cf0a00554349bec73bc7a957a31e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:37:40.612457 18314 ts_tablet_manager.cc:1905] T dd188857a8a744ea86ff35faefe8a310 P 15d7cf0a00554349bec73bc7a957a31e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:37:40.612634 18315 ts_tablet_manager.cc:1905] T dd188857a8a744ea86ff35faefe8a310 P 613199c90419415e9689215ff66f3453: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:37:40.614893 18314 ts_tablet_manager.cc:1918] T dd188857a8a744ea86ff35faefe8a310 P 15d7cf0a00554349bec73bc7a957a31e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:37:40.614996 18314 log.cc:1196] T dd188857a8a744ea86ff35faefe8a310 P 15d7cf0a00554349bec73bc7a957a31e: Deleting WAL directory at /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707036041078-13712-0/minicluster-data/ts-2/wal/wals/dd188857a8a744ea86ff35faefe8a310
I20260929 18:37:40.615283 18314 ts_tablet_manager.cc:1939] T dd188857a8a744ea86ff35faefe8a310 P 15d7cf0a00554349bec73bc7a957a31e: Deleting consensus metadata
I20260929 18:37:40.615861 18315 ts_tablet_manager.cc:1918] T dd188857a8a744ea86ff35faefe8a310 P 613199c90419415e9689215ff66f3453: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:37:40.615957 18315 log.cc:1196] T dd188857a8a744ea86ff35faefe8a310 P 613199c90419415e9689215ff66f3453: Deleting WAL directory at /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707036041078-13712-0/minicluster-data/ts-1/wal/wals/dd188857a8a744ea86ff35faefe8a310
I20260929 18:37:40.616252 18315 ts_tablet_manager.cc:1939] T dd188857a8a744ea86ff35faefe8a310 P 613199c90419415e9689215ff66f3453: Deleting consensus metadata
I20260929 18:37:40.616384 17802 catalog_manager.cc:4841] TS 15d7cf0a00554349bec73bc7a957a31e (127.13.100.3:43687): tablet dd188857a8a744ea86ff35faefe8a310 (table test-table [id=c34f0f963c174fc1ae40f999428534b3]) successfully deleted
W20260929 18:37:40.616938 17805 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.100.1:34957 (error 108)
I20260929 18:37:40.617342 17803 catalog_manager.cc:4841] TS 613199c90419415e9689215ff66f3453 (127.13.100.2:37307): tablet dd188857a8a744ea86ff35faefe8a310 (table test-table [id=c34f0f963c174fc1ae40f999428534b3]) successfully deleted
I20260929 18:37:40.617873 18285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:40.617615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.1:34957 (local address 127.0.0.1:47760)
0929 18:37:40.617721 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:40.617775 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.1:34957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260929 18:37:40.617916 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 18013
W20260929 18:37:40.619467 17805 catalog_manager.cc:4584] TS 1404be66936b491591040170516fcaf8 (127.13.100.1:34957): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet dd188857a8a744ea86ff35faefe8a310: Network error: Client connection negotiation failed: client connection to 127.13.100.1:34957: connect: Connection refused (error 111)
I20260929 18:37:40.623766 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 18146
I20260929 18:37:40.629328 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 17787
2026-09-29T18:37:40Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2379 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-29T18:37:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:37:40Z Disabled control of system clock
I20260929 18:37:40.659050 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707036041078-13712-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707036041078-13712-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707036041078-13712-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.100.62:43967
--webserver_interface=127.13.100.62
--webserver_port=0
--builtin_ntp_servers=127.13.100.20:42847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.100.62:43967
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260929 18:37:40.767129 18321 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:37:40.767377 18321 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:37:40.767414 18321 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:37:40.771306 18321 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:37:40.771359 18321 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:37:40.771374 18321 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:37:40.771391 18321 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:37:40.771406 18321 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:37:40.776394 18321 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.100.20:42847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707036041078-13712-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707036041078-13712-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.100.62:43967
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.100.62:43967
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.100.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18321
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707036041078-13712-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:37:40.777488 18321 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:37:40.779062 18321 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:37:40.787823 18329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:37:40.787883 18326 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:37:40.787823 18327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:37:40.788242 18321 server_base.cc:1034] running on GCE node
I20260929 18:37:40.788884 18321 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:37:40.790148 18321 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:37:40.791361 18321 hybrid_clock.cc:648] HybridClock initialized: now 1790707060791333 us; error 47 us; skew 500 ppm
I20260929 18:37:40.793334 18321 webserver.cc:460] Webserver started at http://127.13.100.62:41885/ using document root <none> and password file <none>
I20260929 18:37:40.794004 18321 fs_manager.cc:359] Metadata directory not provided
I20260929 18:37:40.794097 18321 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:37:40.794335 18321 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:37:40.796416 18321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707036041078-13712-0/minicluster-data/master-0/data/instance:
uuid: "7671cf91394b44598019c1e80ec2447d"
format_stamp: "Formatted at 2026-09-29 18:37:40 on dist-test-slave-j1xs"
server_key: "b2f79ec1ab99716dcd57aa1534eeba45"
server_key_iv: "d2140ac144a328b58d260b2fc3e33557"
server_key_version: "encryptionkey@0"
I20260929 18:37:40.796996 18321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707036041078-13712-0/minicluster-data/master-0/wal/instance:
uuid: "7671cf91394b44598019c1e80ec2447d"
format_stamp: "Formatted at 2026-09-29 18:37:40 on dist-test-slave-j1xs"
server_key: "b2f79ec1ab99716dcd57aa1534eeba45"
server_key_iv: "d2140ac144a328b58d260b2fc3e33557"
server_key_version: "encryptionkey@0"
I20260929 18:37:40.800760 18321 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260929 18:37:40.803284 18335 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:37:40.804509 18321 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 18:37:40.804661 18321 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707036041078-13712-0/minicluster-data/master-0/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707036041078-13712-0/minicluster-data/master-0/wal
uuid: "7671cf91394b44598019c1e80ec2447d"
format_stamp: "Formatted at 2026-09-29 18:37:40 on dist-test-slave-j1xs"
server_key: "b2f79ec1ab99716dcd57aa1534eeba45"
server_key_iv: "d2140ac144a328b58d260b2fc3e33557"
server_key_version: "encryptionkey@0"
I20260929 18:37:40.804777 18321 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707036041078-13712-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707036041078-13712-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707036041078-13712-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:37:40.809242 18321 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:37:40.809845 18321 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:37:40.810067 18321 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:37:40.816855 18321 rpc_server.cc:307] RPC server started. Bound to: 127.13.100.62:43967
I20260929 18:37:40.816864 18387 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.100.62:43967 every 8 connection(s)
I20260929 18:37:40.817756 18321 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
I20260929 18:37:40.820868 18388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:37:40.824393 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 18321
I20260929 18:37:40.824529 13712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707036041078-13712-0/minicluster-data/master-0/wal/instance
I20260929 18:37:40.824801 13712 external_mini_cluster.cc:1428] Setting key 98ddb4eb81b35b47e77d803f1ec4906f
I20260929 18:37:40.826565 18388 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7671cf91394b44598019c1e80ec2447d: Bootstrap starting.
I20260929 18:37:40.828769 18388 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7671cf91394b44598019c1e80ec2447d: Neither blocks nor log segments found. Creating new log.
I20260929 18:37:40.829677 18388 log.cc:824] T 00000000000000000000000000000000 P 7671cf91394b44598019c1e80ec2447d: Log is configured to *not* fsync() on all Append() calls
I20260929 18:37:40.830504 18391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:40.826168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59466 (local address 127.13.100.62:43967)
0929 18:37:40.826840 (+ 672us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:40.826855 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:40.826891 (+ 36us) server_negotiation.cc:407] Connection header received
0929 18:37:40.827529 (+ 638us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:40.827545 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:40.827883 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:40.828234 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:40.829040 (+ 806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:40.829047 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:40.829065 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:37:40.829078 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:40.829147 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:37:40.829171 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:40.829288 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:40.829386 (+ 98us) server_negotiation.cc:299] Negotiation successful
0929 18:37:40.829636 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":128,"threads_started":1}
I20260929 18:37:40.831729 18388 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7671cf91394b44598019c1e80ec2447d: No bootstrap required, opened a new log
I20260929 18:37:40.835336 18388 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7671cf91394b44598019c1e80ec2447d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7671cf91394b44598019c1e80ec2447d" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 43967 } }
I20260929 18:37:40.835544 18388 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7671cf91394b44598019c1e80ec2447d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:37:40.835618 18388 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7671cf91394b44598019c1e80ec2447d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7671cf91394b44598019c1e80ec2447d, State: Initialized, Role: FOLLOWER
I20260929 18:37:40.836059 18388 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7671cf91394b44598019c1e80ec2447d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7671cf91394b44598019c1e80ec2447d" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 43967 } }
I20260929 18:37:40.836198 18388 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7671cf91394b44598019c1e80ec2447d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:37:40.836264 18388 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7671cf91394b44598019c1e80ec2447d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:37:40.836382 18388 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7671cf91394b44598019c1e80ec2447d [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:37:40.837242 18388 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7671cf91394b44598019c1e80ec2447d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7671cf91394b44598019c1e80ec2447d" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 43967 } }
I20260929 18:37:40.837566 18388 leader_election.cc:302] T 00000000000000000000000000000000 P 7671cf91394b44598019c1e80ec2447d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7671cf91394b44598019c1e80ec2447d; no voters:
I20260929 18:37:40.837849 18388 leader_election.cc:288] T 00000000000000000000000000000000 P 7671cf91394b44598019c1e80ec2447d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:37:40.838119 18393 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7671cf91394b44598019c1e80ec2447d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:37:40.838543 18393 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7671cf91394b44598019c1e80ec2447d [term 1 LEADER]: Becoming Leader. State: Replica: 7671cf91394b44598019c1e80ec2447d, State: Running, Role: LEADER
I20260929 18:37:40.838825 18393 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7671cf91394b44598019c1e80ec2447d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7671cf91394b44598019c1e80ec2447d" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 43967 } }
I20260929 18:37:40.839126 18388 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7671cf91394b44598019c1e80ec2447d [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:37:40.840332 18395 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7671cf91394b44598019c1e80ec2447d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7671cf91394b44598019c1e80ec2447d. Latest consensus state: current_term: 1 leader_uuid: "7671cf91394b44598019c1e80ec2447d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7671cf91394b44598019c1e80ec2447d" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 43967 } } }
I20260929 18:37:40.840447 18395 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7671cf91394b44598019c1e80ec2447d [sys.catalog]: This master's current role is: LEADER
I20260929 18:37:40.840749 18394 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7671cf91394b44598019c1e80ec2447d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7671cf91394b44598019c1e80ec2447d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7671cf91394b44598019c1e80ec2447d" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 43967 } } }
I20260929 18:37:40.840842 18394 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7671cf91394b44598019c1e80ec2447d [sys.catalog]: This master's current role is: LEADER
I20260929 18:37:40.841140 18403 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:37:40.844405 18403 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:37:40.849933 18403 catalog_manager.cc:1347] Generated new cluster ID: f1994e47a4724aff8b81b765819c15fc
I20260929 18:37:40.849999 18403 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:37:40.860725 18403 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:37:40.861050 18403 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:37:40.870139 18403 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7671cf91394b44598019c1e80ec2447d: Generated new TSK 0
I20260929 18:37:40.870893 18403 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:37:40.878611 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707036041078-13712-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707036041078-13712-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707036041078-13712-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707036041078-13712-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.100.1:0
--local_ip_for_outbound_sockets=127.13.100.1
--webserver_interface=127.13.100.1
--webserver_port=0
--tserver_master_addrs=127.13.100.62:43967
--builtin_ntp_servers=127.13.100.20:42847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260929 18:37:40.988165 18412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:37:40.988389 18412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:37:40.988426 18412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:37:40.992353 18412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:37:40.992398 18412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:37:40.992450 18412 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.100.1
I20260929 18:37:40.996968 18412 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.100.20:42847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707036041078-13712-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707036041078-13712-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.100.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707036041078-13712-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.100.1
--webserver_port=0
--tserver_master_addrs=127.13.100.62:43967
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.100.1
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707036041078-13712-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:37:40.997840 18412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:37:40.999110 18412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:37:41.008426 18417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:37:41.008497 18412 server_base.cc:1034] running on GCE node
W20260929 18:37:41.008363 18420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:37:41.008363 18418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:37:41.009089 18412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:37:41.009682 18412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:37:41.010897 18412 hybrid_clock.cc:648] HybridClock initialized: now 1790707061010867 us; error 56 us; skew 500 ppm
I20260929 18:37:41.012660 18412 webserver.cc:460] Webserver started at http://127.13.100.1:34437/ using document root <none> and password file <none>
I20260929 18:37:41.013290 18412 fs_manager.cc:359] Metadata directory not provided
I20260929 18:37:41.013378 18412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:37:41.013588 18412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:37:41.015579 18412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707036041078-13712-0/minicluster-data/ts-0/data/instance:
uuid: "d90839b12d054eab98fb8c74e79191b0"
format_stamp: "Formatted at 2026-09-29 18:37:41 on dist-test-slave-j1xs"
server_key: "28c1f28ebdfad31c7dc47e39b15b5939"
server_key_iv: "2bb9bba38d8528216b828a121c96973f"
server_key_version: "encryptionkey@0"
I20260929 18:37:41.016124 18412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707036041078-13712-0/minicluster-data/ts-0/wal/instance:
uuid: "d90839b12d054eab98fb8c74e79191b0"
format_stamp: "Formatted at 2026-09-29 18:37:41 on dist-test-slave-j1xs"
server_key: "28c1f28ebdfad31c7dc47e39b15b5939"
server_key_iv: "2bb9bba38d8528216b828a121c96973f"
server_key_version: "encryptionkey@0"
I20260929 18:37:41.019639 18412 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 18:37:41.021965 18426 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:37:41.023079 18412 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:37:41.023223 18412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707036041078-13712-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707036041078-13712-0/minicluster-data/ts-0/wal
uuid: "d90839b12d054eab98fb8c74e79191b0"
format_stamp: "Formatted at 2026-09-29 18:37:41 on dist-test-slave-j1xs"
server_key: "28c1f28ebdfad31c7dc47e39b15b5939"
server_key_iv: "2bb9bba38d8528216b828a121c96973f"
server_key_version: "encryptionkey@0"
I20260929 18:37:41.023334 18412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707036041078-13712-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707036041078-13712-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707036041078-13712-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:37:41.027562 18412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:37:41.028105 18412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:37:41.028278 18412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:37:41.028789 18412 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:37:41.029611 18412 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:37:41.029680 18412 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:37:41.029747 18412 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:37:41.029799 18412 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:37:41.037855 18412 rpc_server.cc:307] RPC server started. Bound to: 127.13.100.1:36023
I20260929 18:37:41.037935 18539 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.100.1:36023 every 8 connection(s)
I20260929 18:37:41.038738 18412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707036041078-13712-0/minicluster-data/ts-0/data/info.pb
I20260929 18:37:41.044322 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 18412
I20260929 18:37:41.044442 13712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707036041078-13712-0/minicluster-data/ts-0/wal/instance
I20260929 18:37:41.044636 18391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:41.040503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.1:44319 (local address 127.13.100.62:43967)
0929 18:37:41.040634 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:41.040638 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:41.041660 (+ 1022us) server_negotiation.cc:407] Connection header received
0929 18:37:41.042966 (+ 1306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:41.042971 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:41.043020 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:41.043116 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:41.044106 (+ 990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:41.044109 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:41.044111 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:37:41.044123 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:41.044147 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:37:41.044155 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:41.044240 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:41.044461 (+ 221us) server_negotiation.cc:299] Negotiation successful
0929 18:37:41.044509 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 18:37:41.044731 13712 external_mini_cluster.cc:1428] Setting key 02ebd8a497d0f93657ee54139b717313
I20260929 18:37:41.045516 18542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:41.040917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:43967 (local address 127.13.100.1:44319)
0929 18:37:41.041517 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:41.041553 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:41.042738 (+ 1185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:41.043264 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:41.043271 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:41.043351 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:37:41.043837 (+ 486us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:37:41.043838 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:37:41.043875 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:37:41.043936 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:37:41.044251 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:37:41.044260 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:37:41.044267 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 18:37:41.044452 (+ 185us) client_negotiation.cc:240] Negotiation successful
0929 18:37:41.044686 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":108,"threads_started":1}
I20260929 18:37:41.046700 18540 heartbeater.cc:347] Connected to a master server at 127.13.100.62:43967
I20260929 18:37:41.046864 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707036041078-13712-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707036041078-13712-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707036041078-13712-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707036041078-13712-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.100.2:0
--local_ip_for_outbound_sockets=127.13.100.2
--webserver_interface=127.13.100.2
--webserver_port=0
--tserver_master_addrs=127.13.100.62:43967
--builtin_ntp_servers=127.13.100.20:42847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260929 18:37:41.047013 18540 heartbeater.cc:464] Registering TS with master...
I20260929 18:37:41.047402 18540 heartbeater.cc:511] Master 127.13.100.62:43967 requested a full tablet report, sending...
I20260929 18:37:41.048849 18352 ts_manager.cc:194] Registered new tserver with Master: d90839b12d054eab98fb8c74e79191b0 (127.13.100.1:36023)
W20260929 18:37:41.160818 18543 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:37:41.161113 18543 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:37:41.161211 18543 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:37:41.165331 18543 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:37:41.165424 18543 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:37:41.165521 18543 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.100.2
I20260929 18:37:41.170295 18543 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.100.20:42847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707036041078-13712-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707036041078-13712-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.100.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707036041078-13712-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.100.2
--webserver_port=0
--tserver_master_addrs=127.13.100.62:43967
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18543
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.100.2
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707036041078-13712-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:37:41.171406 18543 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:37:41.172880 18543 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:37:41.182257 18548 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:37:41.182260 18551 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:37:41.182277 18549 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:37:41.182408 18543 server_base.cc:1034] running on GCE node
I20260929 18:37:41.182920 18543 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:37:41.183569 18543 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:37:41.184767 18543 hybrid_clock.cc:648] HybridClock initialized: now 1790707061184733 us; error 48 us; skew 500 ppm
I20260929 18:37:41.186817 18543 webserver.cc:460] Webserver started at http://127.13.100.2:42775/ using document root <none> and password file <none>
I20260929 18:37:41.187400 18543 fs_manager.cc:359] Metadata directory not provided
I20260929 18:37:41.187497 18543 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:37:41.187740 18543 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:37:41.189765 18543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707036041078-13712-0/minicluster-data/ts-1/data/instance:
uuid: "c147219b319244cb85870986cde69702"
format_stamp: "Formatted at 2026-09-29 18:37:41 on dist-test-slave-j1xs"
server_key: "eb6feed7856cec0186ef41a552b09d3a"
server_key_iv: "6ac6dadbf0446d50cccdcd4baa73063b"
server_key_version: "encryptionkey@0"
I20260929 18:37:41.190420 18543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707036041078-13712-0/minicluster-data/ts-1/wal/instance:
uuid: "c147219b319244cb85870986cde69702"
format_stamp: "Formatted at 2026-09-29 18:37:41 on dist-test-slave-j1xs"
server_key: "eb6feed7856cec0186ef41a552b09d3a"
server_key_iv: "6ac6dadbf0446d50cccdcd4baa73063b"
server_key_version: "encryptionkey@0"
I20260929 18:37:41.194346 18543 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 18:37:41.196888 18557 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:37:41.198086 18543 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:37:41.198227 18543 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707036041078-13712-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707036041078-13712-0/minicluster-data/ts-1/wal
uuid: "c147219b319244cb85870986cde69702"
format_stamp: "Formatted at 2026-09-29 18:37:41 on dist-test-slave-j1xs"
server_key: "eb6feed7856cec0186ef41a552b09d3a"
server_key_iv: "6ac6dadbf0446d50cccdcd4baa73063b"
server_key_version: "encryptionkey@0"
I20260929 18:37:41.198338 18543 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707036041078-13712-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707036041078-13712-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707036041078-13712-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:37:41.202690 18543 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:37:41.203279 18543 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:37:41.203452 18543 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:37:41.203944 18543 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:37:41.204788 18543 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:37:41.204857 18543 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:37:41.204922 18543 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:37:41.204967 18543 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:37:41.213655 18543 rpc_server.cc:307] RPC server started. Bound to: 127.13.100.2:39381
I20260929 18:37:41.213712 18670 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.100.2:39381 every 8 connection(s)
I20260929 18:37:41.214556 18543 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707036041078-13712-0/minicluster-data/ts-1/data/info.pb
I20260929 18:37:41.220508 18391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:41.216284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.2:36427 (local address 127.13.100.62:43967)
0929 18:37:41.216446 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:41.216450 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:41.217543 (+ 1093us) server_negotiation.cc:407] Connection header received
0929 18:37:41.218934 (+ 1391us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:41.218937 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:41.218974 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:41.219053 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:41.220114 (+ 1061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:41.220118 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:41.220120 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:37:41.220132 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:41.220153 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:37:41.220161 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:41.220241 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:41.220360 (+ 119us) server_negotiation.cc:299] Negotiation successful
0929 18:37:41.220398 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 18:37:41.221395 18673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:41.216664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:43967 (local address 127.13.100.2:36427)
0929 18:37:41.217394 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:41.217433 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:41.218732 (+ 1299us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:41.219246 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:41.219252 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:41.219337 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:37:41.219864 (+ 527us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:37:41.219865 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:37:41.219903 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:37:41.219967 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:37:41.220238 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:37:41.220244 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:37:41.220248 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 18:37:41.220377 (+ 129us) client_negotiation.cc:240] Negotiation successful
0929 18:37:41.220559 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":111,"threads_started":1}
I20260929 18:37:41.222643 18671 heartbeater.cc:347] Connected to a master server at 127.13.100.62:43967
I20260929 18:37:41.222889 18671 heartbeater.cc:464] Registering TS with master...
I20260929 18:37:41.223066 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 18543
I20260929 18:37:41.223183 13712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707036041078-13712-0/minicluster-data/ts-1/wal/instance
I20260929 18:37:41.223379 18671 heartbeater.cc:511] Master 127.13.100.62:43967 requested a full tablet report, sending...
I20260929 18:37:41.223493 13712 external_mini_cluster.cc:1428] Setting key c145c4fdaf46c62bacc56b8f789ab710
I20260929 18:37:41.224620 18351 ts_manager.cc:194] Registered new tserver with Master: c147219b319244cb85870986cde69702 (127.13.100.2:39381)
I20260929 18:37:41.225646 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707036041078-13712-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707036041078-13712-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707036041078-13712-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707036041078-13712-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.100.3:0
--local_ip_for_outbound_sockets=127.13.100.3
--webserver_interface=127.13.100.3
--webserver_port=0
--tserver_master_addrs=127.13.100.62:43967
--builtin_ntp_servers=127.13.100.20:42847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260929 18:37:41.335404 18674 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:37:41.335680 18674 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:37:41.335721 18674 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:37:41.339998 18674 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:37:41.340096 18674 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:37:41.340209 18674 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.100.3
I20260929 18:37:41.345656 18674 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.100.20:42847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707036041078-13712-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707036041078-13712-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.100.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707036041078-13712-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.100.3
--webserver_port=0
--tserver_master_addrs=127.13.100.62:43967
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18674
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.100.3
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707036041078-13712-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:37:41.346812 18674 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:37:41.348428 18674 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:37:41.358507 18682 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:37:41.358487 18680 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:37:41.358574 18679 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:37:41.358575 18674 server_base.cc:1034] running on GCE node
I20260929 18:37:41.359186 18674 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:37:41.359910 18674 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:37:41.361136 18674 hybrid_clock.cc:648] HybridClock initialized: now 1790707061361111 us; error 32 us; skew 500 ppm
I20260929 18:37:41.363148 18674 webserver.cc:460] Webserver started at http://127.13.100.3:42363/ using document root <none> and password file <none>
I20260929 18:37:41.363759 18674 fs_manager.cc:359] Metadata directory not provided
I20260929 18:37:41.363857 18674 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:37:41.364107 18674 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:37:41.366441 18674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707036041078-13712-0/minicluster-data/ts-2/data/instance:
uuid: "f2c54e6a07c449188ae4fe6b9dbb7de9"
format_stamp: "Formatted at 2026-09-29 18:37:41 on dist-test-slave-j1xs"
server_key: "b653c9ec87489b5279ff8f48d8419724"
server_key_iv: "5d778a5f3c0b7cca0c056b400566a14d"
server_key_version: "encryptionkey@0"
I20260929 18:37:41.367038 18674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707036041078-13712-0/minicluster-data/ts-2/wal/instance:
uuid: "f2c54e6a07c449188ae4fe6b9dbb7de9"
format_stamp: "Formatted at 2026-09-29 18:37:41 on dist-test-slave-j1xs"
server_key: "b653c9ec87489b5279ff8f48d8419724"
server_key_iv: "5d778a5f3c0b7cca0c056b400566a14d"
server_key_version: "encryptionkey@0"
I20260929 18:37:41.370885 18674 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 18:37:41.373577 18688 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:37:41.374881 18674 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:37:41.375013 18674 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707036041078-13712-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707036041078-13712-0/minicluster-data/ts-2/wal
uuid: "f2c54e6a07c449188ae4fe6b9dbb7de9"
format_stamp: "Formatted at 2026-09-29 18:37:41 on dist-test-slave-j1xs"
server_key: "b653c9ec87489b5279ff8f48d8419724"
server_key_iv: "5d778a5f3c0b7cca0c056b400566a14d"
server_key_version: "encryptionkey@0"
I20260929 18:37:41.375144 18674 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707036041078-13712-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707036041078-13712-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707036041078-13712-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:37:41.379710 18674 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:37:41.380304 18674 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:37:41.380481 18674 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:37:41.381004 18674 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:37:41.382031 18674 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:37:41.382123 18674 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:37:41.382194 18674 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:37:41.382246 18674 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:37:41.390810 18674 rpc_server.cc:307] RPC server started. Bound to: 127.13.100.3:35583
I20260929 18:37:41.390887 18801 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.100.3:35583 every 8 connection(s)
I20260929 18:37:41.391755 18674 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707036041078-13712-0/minicluster-data/ts-2/data/info.pb
I20260929 18:37:41.397812 18391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:41.393642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.3:45499 (local address 127.13.100.62:43967)
0929 18:37:41.393779 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:41.393783 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:41.394879 (+ 1096us) server_negotiation.cc:407] Connection header received
0929 18:37:41.396184 (+ 1305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:41.396188 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:41.396232 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:41.396338 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:41.397351 (+ 1013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:41.397354 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:41.397357 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:37:41.397369 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:41.397392 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:37:41.397405 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:41.397501 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:41.397639 (+ 138us) server_negotiation.cc:299] Negotiation successful
0929 18:37:41.397682 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 18:37:41.398674 18804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:41.394089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:43967 (local address 127.13.100.3:45499)
0929 18:37:41.394737 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:41.394775 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:41.395964 (+ 1189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:41.396497 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:41.396503 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:41.396585 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:37:41.397107 (+ 522us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:37:41.397108 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:37:41.397144 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:37:41.397209 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:37:41.397488 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:37:41.397495 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:37:41.397500 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 18:37:41.397689 (+ 189us) client_negotiation.cc:240] Negotiation successful
0929 18:37:41.397930 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":112,"threads_started":1}
I20260929 18:37:41.399806 18802 heartbeater.cc:347] Connected to a master server at 127.13.100.62:43967
I20260929 18:37:41.400024 18802 heartbeater.cc:464] Registering TS with master...
I20260929 18:37:41.400409 18802 heartbeater.cc:511] Master 127.13.100.62:43967 requested a full tablet report, sending...
I20260929 18:37:41.401427 18351 ts_manager.cc:194] Registered new tserver with Master: f2c54e6a07c449188ae4fe6b9dbb7de9 (127.13.100.3:35583)
I20260929 18:37:41.401957 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 18674
I20260929 18:37:41.402068 13712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707036041078-13712-0/minicluster-data/ts-2/wal/instance
I20260929 18:37:41.402351 13712 external_mini_cluster.cc:1428] Setting key 9c79e3c6ad62b17853d5a562f26bbd0e
I20260929 18:37:41.404274 13712 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 18:37:41.407433 18391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:41.406036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38522 (local address 127.13.100.62:43967)
0929 18:37:41.406177 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:41.406180 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:41.406352 (+ 172us) server_negotiation.cc:407] Connection header received
0929 18:37:41.406440 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:41.406442 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:41.406469 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:41.406532 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:41.407120 (+ 588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:41.407123 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:41.407125 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:37:41.407148 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:41.407164 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:37:41.407171 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:41.407236 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:41.407313 (+ 77us) server_negotiation.cc:299] Negotiation successful
0929 18:37:41.407341 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 18:37:41.410235 18351 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38522:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:37:41.413121 18351 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:37:41.424081 18813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:41.421081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.2:39381 (local address 127.0.0.1:38548)
0929 18:37:41.421554 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:41.421597 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:41.421692 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:41.422425 (+ 733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:41.422430 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:41.422502 (+ 72us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:37:41.423274 (+ 772us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:37:41.423276 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:37:41.423295 (+ 19us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:37:41.423360 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:37:41.423722 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:37:41.423739 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:37:41.423747 (+ 8us) client_negotiation.cc:769] Sending connection context
0929 18:37:41.423877 (+ 130us) client_negotiation.cc:240] Negotiation successful
0929 18:37:41.423944 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":75,"threads_started":1}
I20260929 18:37:41.424101 18814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:41.421231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38548 (local address 127.13.100.2:39381)
0929 18:37:41.421977 (+ 746us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:41.421989 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:41.422015 (+ 26us) server_negotiation.cc:407] Connection header received
0929 18:37:41.422066 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:41.422071 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:41.422256 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:41.422403 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:41.423506 (+ 1103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:41.423514 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:41.423528 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:37:41.423543 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:41.423581 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:37:41.423618 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:41.423733 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:41.423906 (+ 173us) server_negotiation.cc:299] Negotiation successful
0929 18:37:41.423973 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":628,"thread_start_us":130,"threads_started":1}
I20260929 18:37:41.425091 18815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:41.421277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.3:35583 (local address 127.0.0.1:43300)
0929 18:37:41.421608 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:41.421622 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:41.421724 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:41.423406 (+ 1682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:41.423409 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:41.423424 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:37:41.424472 (+ 1048us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:37:41.424474 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:37:41.424484 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:37:41.424515 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:37:41.424849 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:37:41.424851 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:37:41.424854 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 18:37:41.424930 (+ 76us) client_negotiation.cc:240] Negotiation successful
0929 18:37:41.424964 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":103,"threads_started":1}
I20260929 18:37:41.425249 18816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:41.421764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.1:36023 (local address 127.0.0.1:46418)
0929 18:37:41.422001 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:41.422015 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:41.422090 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:41.423366 (+ 1276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:41.423368 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:41.423382 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:37:41.424106 (+ 724us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:37:41.424110 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:37:41.424614 (+ 504us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:37:41.424631 (+ 17us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:37:41.425030 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:37:41.425033 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:37:41.425036 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 18:37:41.425105 (+ 69us) client_negotiation.cc:240] Negotiation successful
0929 18:37:41.425135 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":120,"threads_started":1}
I20260929 18:37:41.425556 18817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:41.421928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46418 (local address 127.13.100.1:36023)
0929 18:37:41.422358 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:41.422371 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:41.422393 (+ 22us) server_negotiation.cc:407] Connection header received
0929 18:37:41.422439 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:41.422444 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:41.422609 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:41.422754 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:41.424779 (+ 2025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:41.424788 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:41.424802 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:37:41.424819 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:41.424863 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:37:41.424892 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:41.424987 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:41.425339 (+ 352us) server_negotiation.cc:299] Negotiation successful
0929 18:37:41.425410 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":133,"threads_started":1}
I20260929 18:37:41.425856 18818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:41.422166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43300 (local address 127.13.100.3:35583)
0929 18:37:41.422955 (+ 789us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:41.422967 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:41.422994 (+ 27us) server_negotiation.cc:407] Connection header received
0929 18:37:41.423047 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:41.423054 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:41.423221 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:41.423399 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:41.424639 (+ 1240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:41.424643 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:41.424651 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:37:41.424661 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:41.424722 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:37:41.424753 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:41.424864 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:41.424945 (+ 81us) server_negotiation.cc:299] Negotiation successful
0929 18:37:41.425018 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":684,"thread_start_us":131,"threads_started":1}
I20260929 18:37:41.427273 18605 tablet_service.cc:1467] Processing CreateTablet for tablet 0398e9d556214dbeb08579ff737a362e (DEFAULT_TABLE table=test-table [id=0bcaf16464b94c04bfd19b094dfffa0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:37:41.428254 18605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0398e9d556214dbeb08579ff737a362e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:37:41.428336 18736 tablet_service.cc:1467] Processing CreateTablet for tablet 0398e9d556214dbeb08579ff737a362e (DEFAULT_TABLE table=test-table [id=0bcaf16464b94c04bfd19b094dfffa0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:37:41.428248 18474 tablet_service.cc:1467] Processing CreateTablet for tablet 0398e9d556214dbeb08579ff737a362e (DEFAULT_TABLE table=test-table [id=0bcaf16464b94c04bfd19b094dfffa0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:37:41.429231 18736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0398e9d556214dbeb08579ff737a362e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:37:41.429239 18474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0398e9d556214dbeb08579ff737a362e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:37:41.434809 18819 tablet_bootstrap.cc:492] T 0398e9d556214dbeb08579ff737a362e P c147219b319244cb85870986cde69702: Bootstrap starting.
I20260929 18:37:41.436419 18820 tablet_bootstrap.cc:492] T 0398e9d556214dbeb08579ff737a362e P f2c54e6a07c449188ae4fe6b9dbb7de9: Bootstrap starting.
I20260929 18:37:41.436789 18821 tablet_bootstrap.cc:492] T 0398e9d556214dbeb08579ff737a362e P d90839b12d054eab98fb8c74e79191b0: Bootstrap starting.
I20260929 18:37:41.437181 18819 tablet_bootstrap.cc:654] T 0398e9d556214dbeb08579ff737a362e P c147219b319244cb85870986cde69702: Neither blocks nor log segments found. Creating new log.
I20260929 18:37:41.438328 18819 log.cc:824] T 0398e9d556214dbeb08579ff737a362e P c147219b319244cb85870986cde69702: Log is configured to *not* fsync() on all Append() calls
I20260929 18:37:41.438534 18820 tablet_bootstrap.cc:654] T 0398e9d556214dbeb08579ff737a362e P f2c54e6a07c449188ae4fe6b9dbb7de9: Neither blocks nor log segments found. Creating new log.
I20260929 18:37:41.438668 18821 tablet_bootstrap.cc:654] T 0398e9d556214dbeb08579ff737a362e P d90839b12d054eab98fb8c74e79191b0: Neither blocks nor log segments found. Creating new log.
I20260929 18:37:41.439436 18820 log.cc:824] T 0398e9d556214dbeb08579ff737a362e P f2c54e6a07c449188ae4fe6b9dbb7de9: Log is configured to *not* fsync() on all Append() calls
I20260929 18:37:41.439433 18821 log.cc:824] T 0398e9d556214dbeb08579ff737a362e P d90839b12d054eab98fb8c74e79191b0: Log is configured to *not* fsync() on all Append() calls
I20260929 18:37:41.440438 18819 tablet_bootstrap.cc:492] T 0398e9d556214dbeb08579ff737a362e P c147219b319244cb85870986cde69702: No bootstrap required, opened a new log
I20260929 18:37:41.440649 18819 ts_tablet_manager.cc:1397] T 0398e9d556214dbeb08579ff737a362e P c147219b319244cb85870986cde69702: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 18:37:41.441375 18821 tablet_bootstrap.cc:492] T 0398e9d556214dbeb08579ff737a362e P d90839b12d054eab98fb8c74e79191b0: No bootstrap required, opened a new log
I20260929 18:37:41.441548 18821 ts_tablet_manager.cc:1397] T 0398e9d556214dbeb08579ff737a362e P d90839b12d054eab98fb8c74e79191b0: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 18:37:41.441804 18820 tablet_bootstrap.cc:492] T 0398e9d556214dbeb08579ff737a362e P f2c54e6a07c449188ae4fe6b9dbb7de9: No bootstrap required, opened a new log
I20260929 18:37:41.441992 18820 ts_tablet_manager.cc:1397] T 0398e9d556214dbeb08579ff737a362e P f2c54e6a07c449188ae4fe6b9dbb7de9: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260929 18:37:41.444666 18819 raft_consensus.cc:348] T 0398e9d556214dbeb08579ff737a362e P c147219b319244cb85870986cde69702 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c147219b319244cb85870986cde69702" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 39381 } } peers { permanent_uuid: "f2c54e6a07c449188ae4fe6b9dbb7de9" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 35583 } } peers { permanent_uuid: "d90839b12d054eab98fb8c74e79191b0" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 36023 } }
I20260929 18:37:41.444908 18819 raft_consensus.cc:374] T 0398e9d556214dbeb08579ff737a362e P c147219b319244cb85870986cde69702 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:37:41.444964 18819 raft_consensus.cc:729] T 0398e9d556214dbeb08579ff737a362e P c147219b319244cb85870986cde69702 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c147219b319244cb85870986cde69702, State: Initialized, Role: FOLLOWER
I20260929 18:37:41.445438 18819 consensus_queue.cc:260] T 0398e9d556214dbeb08579ff737a362e P c147219b319244cb85870986cde69702 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c147219b319244cb85870986cde69702" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 39381 } } peers { permanent_uuid: "f2c54e6a07c449188ae4fe6b9dbb7de9" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 35583 } } peers { permanent_uuid: "d90839b12d054eab98fb8c74e79191b0" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 36023 } }
I20260929 18:37:41.446126 18671 heartbeater.cc:503] Master 127.13.100.62:43967 was elected leader, sending a full tablet report...
I20260929 18:37:41.446175 18820 raft_consensus.cc:348] T 0398e9d556214dbeb08579ff737a362e P f2c54e6a07c449188ae4fe6b9dbb7de9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c147219b319244cb85870986cde69702" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 39381 } } peers { permanent_uuid: "f2c54e6a07c449188ae4fe6b9dbb7de9" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 35583 } } peers { permanent_uuid: "d90839b12d054eab98fb8c74e79191b0" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 36023 } }
I20260929 18:37:41.446175 18821 raft_consensus.cc:348] T 0398e9d556214dbeb08579ff737a362e P d90839b12d054eab98fb8c74e79191b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c147219b319244cb85870986cde69702" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 39381 } } peers { permanent_uuid: "f2c54e6a07c449188ae4fe6b9dbb7de9" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 35583 } } peers { permanent_uuid: "d90839b12d054eab98fb8c74e79191b0" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 36023 } }
I20260929 18:37:41.446422 18821 raft_consensus.cc:374] T 0398e9d556214dbeb08579ff737a362e P d90839b12d054eab98fb8c74e79191b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:37:41.446422 18820 raft_consensus.cc:374] T 0398e9d556214dbeb08579ff737a362e P f2c54e6a07c449188ae4fe6b9dbb7de9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:37:41.446478 18820 raft_consensus.cc:729] T 0398e9d556214dbeb08579ff737a362e P f2c54e6a07c449188ae4fe6b9dbb7de9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2c54e6a07c449188ae4fe6b9dbb7de9, State: Initialized, Role: FOLLOWER
I20260929 18:37:41.446486 18821 raft_consensus.cc:729] T 0398e9d556214dbeb08579ff737a362e P d90839b12d054eab98fb8c74e79191b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d90839b12d054eab98fb8c74e79191b0, State: Initialized, Role: FOLLOWER
I20260929 18:37:41.446893 18820 consensus_queue.cc:260] T 0398e9d556214dbeb08579ff737a362e P f2c54e6a07c449188ae4fe6b9dbb7de9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c147219b319244cb85870986cde69702" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 39381 } } peers { permanent_uuid: "f2c54e6a07c449188ae4fe6b9dbb7de9" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 35583 } } peers { permanent_uuid: "d90839b12d054eab98fb8c74e79191b0" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 36023 } }
I20260929 18:37:41.446893 18821 consensus_queue.cc:260] T 0398e9d556214dbeb08579ff737a362e P d90839b12d054eab98fb8c74e79191b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c147219b319244cb85870986cde69702" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 39381 } } peers { permanent_uuid: "f2c54e6a07c449188ae4fe6b9dbb7de9" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 35583 } } peers { permanent_uuid: "d90839b12d054eab98fb8c74e79191b0" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 36023 } }
I20260929 18:37:41.447846 18820 ts_tablet_manager.cc:1428] T 0398e9d556214dbeb08579ff737a362e P f2c54e6a07c449188ae4fe6b9dbb7de9: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260929 18:37:41.447846 18821 ts_tablet_manager.cc:1428] T 0398e9d556214dbeb08579ff737a362e P d90839b12d054eab98fb8c74e79191b0: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 18:37:41.447875 18819 ts_tablet_manager.cc:1428] T 0398e9d556214dbeb08579ff737a362e P c147219b319244cb85870986cde69702: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260929 18:37:41.448175 18802 heartbeater.cc:503] Master 127.13.100.62:43967 was elected leader, sending a full tablet report...
I20260929 18:37:41.448176 18540 heartbeater.cc:503] Master 127.13.100.62:43967 was elected leader, sending a full tablet report...
W20260929 18:37:41.465427 18672 tablet.cc:2369] T 0398e9d556214dbeb08579ff737a362e P c147219b319244cb85870986cde69702: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 18:37:41.539918 18541 tablet.cc:2369] T 0398e9d556214dbeb08579ff737a362e P d90839b12d054eab98fb8c74e79191b0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:37:41.576011 18825 raft_consensus.cc:482] T 0398e9d556214dbeb08579ff737a362e P c147219b319244cb85870986cde69702 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:37:41.576202 18825 raft_consensus.cc:504] T 0398e9d556214dbeb08579ff737a362e P c147219b319244cb85870986cde69702 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c147219b319244cb85870986cde69702" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 39381 } } peers { permanent_uuid: "f2c54e6a07c449188ae4fe6b9dbb7de9" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 35583 } } peers { permanent_uuid: "d90839b12d054eab98fb8c74e79191b0" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 36023 } }
I20260929 18:37:41.577275 18825 leader_election.cc:288] T 0398e9d556214dbeb08579ff737a362e P c147219b319244cb85870986cde69702 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f2c54e6a07c449188ae4fe6b9dbb7de9 (127.13.100.3:35583), d90839b12d054eab98fb8c74e79191b0 (127.13.100.1:36023)
I20260929 18:37:41.579208 18818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:41.577579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.2:53793 (local address 127.13.100.3:35583)
0929 18:37:41.577731 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:41.577736 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:41.577751 (+ 15us) server_negotiation.cc:407] Connection header received
0929 18:37:41.577904 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:41.577910 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:41.577955 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:41.578039 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:41.578847 (+ 808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:41.578850 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:41.578853 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:37:41.578866 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:41.578888 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:37:41.578898 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:41.579019 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:41.579054 (+ 35us) server_negotiation.cc:299] Negotiation successful
0929 18:37:41.579092 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 18:37:41.579213 18673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:41.577469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.3:35583 (local address 127.13.100.2:53793)
0929 18:37:41.577614 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:41.577647 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:41.577736 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:41.578071 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:41.578074 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:41.578104 (+ 30us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:37:41.578685 (+ 581us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:37:41.578686 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:37:41.578694 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:37:41.578740 (+ 46us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:37:41.578973 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:37:41.578978 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:37:41.578982 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 18:37:41.579053 (+ 71us) client_negotiation.cc:240] Negotiation successful
0929 18:37:41.579116 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260929 18:37:41.579886 18756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0398e9d556214dbeb08579ff737a362e" candidate_uuid: "c147219b319244cb85870986cde69702" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2c54e6a07c449188ae4fe6b9dbb7de9" is_pre_election: true
I20260929 18:37:41.580026 18817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:41.577974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.2:49889 (local address 127.13.100.1:36023)
0929 18:37:41.578097 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:41.578102 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:41.578213 (+ 111us) server_negotiation.cc:407] Connection header received
0929 18:37:41.578373 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:41.578375 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:41.578408 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:41.578470 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:41.579635 (+ 1165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:41.579638 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:41.579641 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:37:41.579659 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:41.579679 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:37:41.579687 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:41.579784 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:41.579860 (+ 76us) server_negotiation.cc:299] Negotiation successful
0929 18:37:41.579896 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 18:37:41.580113 18828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:41.577757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.1:36023 (local address 127.13.100.2:49889)
0929 18:37:41.578096 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:41.578109 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:41.578244 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:41.578515 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:41.578519 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:41.578537 (+ 18us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:37:41.579391 (+ 854us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:37:41.579393 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:37:41.579403 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:37:41.579493 (+ 90us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:37:41.579780 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:37:41.579783 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:37:41.579786 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 18:37:41.579862 (+ 76us) client_negotiation.cc:240] Negotiation successful
0929 18:37:41.579896 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":151,"threads_started":1}
I20260929 18:37:41.580237 18756 raft_consensus.cc:2454] T 0398e9d556214dbeb08579ff737a362e P f2c54e6a07c449188ae4fe6b9dbb7de9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c147219b319244cb85870986cde69702 in term 0.
I20260929 18:37:41.580739 18560 leader_election.cc:302] T 0398e9d556214dbeb08579ff737a362e P c147219b319244cb85870986cde69702 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c147219b319244cb85870986cde69702, f2c54e6a07c449188ae4fe6b9dbb7de9; no voters:
I20260929 18:37:41.580936 18494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0398e9d556214dbeb08579ff737a362e" candidate_uuid: "c147219b319244cb85870986cde69702" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d90839b12d054eab98fb8c74e79191b0" is_pre_election: true
I20260929 18:37:41.581033 18825 raft_consensus.cc:2781] T 0398e9d556214dbeb08579ff737a362e P c147219b319244cb85870986cde69702 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:37:41.581121 18825 raft_consensus.cc:482] T 0398e9d556214dbeb08579ff737a362e P c147219b319244cb85870986cde69702 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:37:41.581195 18825 raft_consensus.cc:3037] T 0398e9d556214dbeb08579ff737a362e P c147219b319244cb85870986cde69702 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:37:41.581243 18494 raft_consensus.cc:2454] T 0398e9d556214dbeb08579ff737a362e P d90839b12d054eab98fb8c74e79191b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c147219b319244cb85870986cde69702 in term 0.
I20260929 18:37:41.582417 18825 raft_consensus.cc:504] T 0398e9d556214dbeb08579ff737a362e P c147219b319244cb85870986cde69702 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c147219b319244cb85870986cde69702" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 39381 } } peers { permanent_uuid: "f2c54e6a07c449188ae4fe6b9dbb7de9" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 35583 } } peers { permanent_uuid: "d90839b12d054eab98fb8c74e79191b0" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 36023 } }
I20260929 18:37:41.582791 18825 leader_election.cc:288] T 0398e9d556214dbeb08579ff737a362e P c147219b319244cb85870986cde69702 [CANDIDATE]: Term 1 election: Requested vote from peers f2c54e6a07c449188ae4fe6b9dbb7de9 (127.13.100.3:35583), d90839b12d054eab98fb8c74e79191b0 (127.13.100.1:36023)
I20260929 18:37:41.583146 18756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0398e9d556214dbeb08579ff737a362e" candidate_uuid: "c147219b319244cb85870986cde69702" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2c54e6a07c449188ae4fe6b9dbb7de9"
I20260929 18:37:41.583138 18494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0398e9d556214dbeb08579ff737a362e" candidate_uuid: "c147219b319244cb85870986cde69702" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d90839b12d054eab98fb8c74e79191b0"
I20260929 18:37:41.583266 18494 raft_consensus.cc:3037] T 0398e9d556214dbeb08579ff737a362e P d90839b12d054eab98fb8c74e79191b0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:37:41.583266 18756 raft_consensus.cc:3037] T 0398e9d556214dbeb08579ff737a362e P f2c54e6a07c449188ae4fe6b9dbb7de9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:37:41.584564 18494 raft_consensus.cc:2454] T 0398e9d556214dbeb08579ff737a362e P d90839b12d054eab98fb8c74e79191b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c147219b319244cb85870986cde69702 in term 1.
I20260929 18:37:41.584585 18756 raft_consensus.cc:2454] T 0398e9d556214dbeb08579ff737a362e P f2c54e6a07c449188ae4fe6b9dbb7de9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c147219b319244cb85870986cde69702 in term 1.
I20260929 18:37:41.584846 18560 leader_election.cc:302] T 0398e9d556214dbeb08579ff737a362e P c147219b319244cb85870986cde69702 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c147219b319244cb85870986cde69702, d90839b12d054eab98fb8c74e79191b0; no voters:
I20260929 18:37:41.584998 18825 raft_consensus.cc:2781] T 0398e9d556214dbeb08579ff737a362e P c147219b319244cb85870986cde69702 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:37:41.585214 18825 raft_consensus.cc:686] T 0398e9d556214dbeb08579ff737a362e P c147219b319244cb85870986cde69702 [term 1 LEADER]: Becoming Leader. State: Replica: c147219b319244cb85870986cde69702, State: Running, Role: LEADER
I20260929 18:37:41.585537 18825 consensus_queue.cc:237] T 0398e9d556214dbeb08579ff737a362e P c147219b319244cb85870986cde69702 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c147219b319244cb85870986cde69702" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 39381 } } peers { permanent_uuid: "f2c54e6a07c449188ae4fe6b9dbb7de9" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 35583 } } peers { permanent_uuid: "d90839b12d054eab98fb8c74e79191b0" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 36023 } }
I20260929 18:37:41.588912 18352 catalog_manager.cc:5495] T 0398e9d556214dbeb08579ff737a362e P c147219b319244cb85870986cde69702 reported cstate change: term changed from 0 to 1, leader changed from <none> to c147219b319244cb85870986cde69702 (127.13.100.2). New cstate: current_term: 1 leader_uuid: "c147219b319244cb85870986cde69702" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c147219b319244cb85870986cde69702" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 39381 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f2c54e6a07c449188ae4fe6b9dbb7de9" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 35583 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d90839b12d054eab98fb8c74e79191b0" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 36023 } health_report { overall_health: UNKNOWN } } }
I20260929 18:37:41.616515 18814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:41.614807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38552 (local address 127.13.100.2:39381)
0929 18:37:41.614941 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:41.614948 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:41.614997 (+ 49us) server_negotiation.cc:407] Connection header received
0929 18:37:41.615212 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:41.615215 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:41.615276 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:41.615372 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:41.616114 (+ 742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:41.616118 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:41.616121 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:37:41.616133 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:41.616153 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:37:41.616169 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:41.616264 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:41.616343 (+ 79us) server_negotiation.cc:299] Negotiation successful
0929 18:37:41.616380 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 18:37:41.622349 18756 raft_consensus.cc:1261] T 0398e9d556214dbeb08579ff737a362e P f2c54e6a07c449188ae4fe6b9dbb7de9 [term 1 FOLLOWER]: Refusing update from remote peer c147219b319244cb85870986cde69702: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:37:41.622349 18494 raft_consensus.cc:1261] T 0398e9d556214dbeb08579ff737a362e P d90839b12d054eab98fb8c74e79191b0 [term 1 FOLLOWER]: Refusing update from remote peer c147219b319244cb85870986cde69702: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:37:41.623086 18825 consensus_queue.cc:1035] T 0398e9d556214dbeb08579ff737a362e P c147219b319244cb85870986cde69702 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d90839b12d054eab98fb8c74e79191b0" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 36023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:37:41.623334 18829 consensus_queue.cc:1035] T 0398e9d556214dbeb08579ff737a362e P c147219b319244cb85870986cde69702 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2c54e6a07c449188ae4fe6b9dbb7de9" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 35583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:37:41.629981 18832 mvcc.cc:205] Tried to move back new op lower bound from 7334736124397445120 to 7334736124257153024. Current Snapshot: MvccSnapshot[applied={T|T < 7334736124397445120}]
I20260929 18:37:41.632846 18833 mvcc.cc:205] Tried to move back new op lower bound from 7334736124397445120 to 7334736124257153024. Current Snapshot: MvccSnapshot[applied={T|T < 7334736124397445120}]
I20260929 18:37:41.635245 18834 mvcc.cc:205] Tried to move back new op lower bound from 7334736124397445120 to 7334736124257153024. Current Snapshot: MvccSnapshot[applied={T|T < 7334736124397445120 or (T in {7334736124397445120})}]
I20260929 18:37:41.639565 18351 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:38522:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:37:41.639724 18351 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:38522:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:37:41.642194 18351 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7671cf91394b44598019c1e80ec2447d: Sending DeleteTablet for 3 replicas of tablet 0398e9d556214dbeb08579ff737a362e
I20260929 18:37:41.642903 18605 tablet_service.cc:1514] Processing DeleteTablet for tablet 0398e9d556214dbeb08579ff737a362e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:37:41 UTC) from {username='slave'} at 127.0.0.1:38548
I20260929 18:37:41.643697 18842 tablet_replica.cc:330] T 0398e9d556214dbeb08579ff737a362e P c147219b319244cb85870986cde69702: stopping tablet replica
I20260929 18:37:41.643648 18736 tablet_service.cc:1514] Processing DeleteTablet for tablet 0398e9d556214dbeb08579ff737a362e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:37:41 UTC) from {username='slave'} at 127.0.0.1:43300
I20260929 18:37:41.643671 18474 tablet_service.cc:1514] Processing DeleteTablet for tablet 0398e9d556214dbeb08579ff737a362e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:37:41 UTC) from {username='slave'} at 127.0.0.1:46418
I20260929 18:37:41.643991 18842 raft_consensus.cc:2229] T 0398e9d556214dbeb08579ff737a362e P c147219b319244cb85870986cde69702 [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:37:41.644351 18842 raft_consensus.cc:2258] T 0398e9d556214dbeb08579ff737a362e P c147219b319244cb85870986cde69702 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:37:41.644600 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 18412
I20260929 18:37:41.644820 18843 tablet_replica.cc:330] T 0398e9d556214dbeb08579ff737a362e P f2c54e6a07c449188ae4fe6b9dbb7de9: stopping tablet replica
I20260929 18:37:41.645210 18843 raft_consensus.cc:2229] T 0398e9d556214dbeb08579ff737a362e P f2c54e6a07c449188ae4fe6b9dbb7de9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:37:41.645509 18843 raft_consensus.cc:2258] T 0398e9d556214dbeb08579ff737a362e P f2c54e6a07c449188ae4fe6b9dbb7de9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:37:41.646997 18843 ts_tablet_manager.cc:1905] T 0398e9d556214dbeb08579ff737a362e P f2c54e6a07c449188ae4fe6b9dbb7de9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:37:41.647356 18842 ts_tablet_manager.cc:1905] T 0398e9d556214dbeb08579ff737a362e P c147219b319244cb85870986cde69702: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:37:41.650027 18842 ts_tablet_manager.cc:1918] T 0398e9d556214dbeb08579ff737a362e P c147219b319244cb85870986cde69702: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:37:41.650123 18842 log.cc:1196] T 0398e9d556214dbeb08579ff737a362e P c147219b319244cb85870986cde69702: Deleting WAL directory at /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707036041078-13712-0/minicluster-data/ts-1/wal/wals/0398e9d556214dbeb08579ff737a362e
I20260929 18:37:41.650276 18843 ts_tablet_manager.cc:1918] T 0398e9d556214dbeb08579ff737a362e P f2c54e6a07c449188ae4fe6b9dbb7de9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:37:41.650377 18843 log.cc:1196] T 0398e9d556214dbeb08579ff737a362e P f2c54e6a07c449188ae4fe6b9dbb7de9: Deleting WAL directory at /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707036041078-13712-0/minicluster-data/ts-2/wal/wals/0398e9d556214dbeb08579ff737a362e
I20260929 18:37:41.650463 18842 ts_tablet_manager.cc:1939] T 0398e9d556214dbeb08579ff737a362e P c147219b319244cb85870986cde69702: Deleting consensus metadata
I20260929 18:37:41.650718 18843 ts_tablet_manager.cc:1939] T 0398e9d556214dbeb08579ff737a362e P f2c54e6a07c449188ae4fe6b9dbb7de9: Deleting consensus metadata
I20260929 18:37:41.651839 18338 catalog_manager.cc:4841] TS f2c54e6a07c449188ae4fe6b9dbb7de9 (127.13.100.3:35583): tablet 0398e9d556214dbeb08579ff737a362e (table test-table [id=0bcaf16464b94c04bfd19b094dfffa0a]) successfully deleted
I20260929 18:37:41.651803 18339 catalog_manager.cc:4841] TS c147219b319244cb85870986cde69702 (127.13.100.2:39381): tablet 0398e9d556214dbeb08579ff737a362e (table test-table [id=0bcaf16464b94c04bfd19b094dfffa0a]) successfully deleted
W20260929 18:37:41.654611 18338 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.100.1:36023 (error 108)
I20260929 18:37:41.655474 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 18543
I20260929 18:37:41.655773 18816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:41.655418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.1:36023 (local address 127.0.0.1:46428)
0929 18:37:41.655583 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:41.655666 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.1:36023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260929 18:37:41.656564 18338 catalog_manager.cc:4584] TS d90839b12d054eab98fb8c74e79191b0 (127.13.100.1:36023): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0398e9d556214dbeb08579ff737a362e: Network error: Client connection negotiation failed: client connection to 127.13.100.1:36023: connect: Connection refused (error 111)
I20260929 18:37:41.660836 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 18674
I20260929 18:37:41.666376 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 18321
2026-09-29T18:37:41Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1036 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260929 18:37:41.687671 13712 mini_oidc.cc:152] Starting JWKS server
I20260929 18:37:41.689205 13712 webserver.cc:460] Webserver started at https://127.0.0.1:43375/ using document root <none> and password file <none>
I20260929 18:37:41.689384 13712 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 18:37:41.689659 13712 webserver.cc:460] Webserver started at http://127.0.0.1:41745/ using document root <none> and password file <none>
2026-09-29T18:37:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:37:41Z Disabled control of system clock
I20260929 18:37:41.698114 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790707036041078-13712-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790707036041078-13712-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790707036041078-13712-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.100.62:41087
--webserver_interface=127.13.100.62
--webserver_port=0
--builtin_ntp_servers=127.13.100.20:46691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.100.62:41087
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43375/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790707036041078-13712-0/testchainca.cert with env {}
W20260929 18:37:41.804685 18851 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:37:41.804965 18851 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:37:41.805006 18851 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:37:41.809118 18851 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:37:41.809194 18851 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:37:41.809209 18851 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:37:41.809227 18851 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:37:41.809243 18851 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 18:37:41.809259 18851 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 18:37:41.809321 18851 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43375/jwks.json
I20260929 18:37:41.814816 18851 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.100.20:46691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790707036041078-13712-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790707036041078-13712-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.100.62:41087
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.100.62:41087
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43375/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.100.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790707036041078-13712-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18851
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790707036041078-13712-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:37:41.815893 18851 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:37:41.816888 18851 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:37:41.825174 18857 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:37:41.825218 18859 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:37:41.825209 18856 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:37:41.825448 18851 server_base.cc:1034] running on GCE node
I20260929 18:37:41.826053 18851 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:37:41.827172 18851 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:37:41.828383 18851 hybrid_clock.cc:648] HybridClock initialized: now 1790707061828352 us; error 46 us; skew 500 ppm
I20260929 18:37:41.830329 18851 webserver.cc:460] Webserver started at http://127.13.100.62:38465/ using document root <none> and password file <none>
I20260929 18:37:41.830885 18851 fs_manager.cc:359] Metadata directory not provided
I20260929 18:37:41.830977 18851 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:37:41.831203 18851 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:37:41.833189 18851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790707036041078-13712-0/minicluster-data/master-0/data/instance:
uuid: "9cfcc8219ed245d5baa67c8a06bac2bb"
format_stamp: "Formatted at 2026-09-29 18:37:41 on dist-test-slave-j1xs"
server_key: "bc4ee7ed931e5dfcd33d675ddb12e53b"
server_key_iv: "4971831d703fc8083870b1ac1c69603c"
server_key_version: "encryptionkey@0"
I20260929 18:37:41.833748 18851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790707036041078-13712-0/minicluster-data/master-0/wal/instance:
uuid: "9cfcc8219ed245d5baa67c8a06bac2bb"
format_stamp: "Formatted at 2026-09-29 18:37:41 on dist-test-slave-j1xs"
server_key: "bc4ee7ed931e5dfcd33d675ddb12e53b"
server_key_iv: "4971831d703fc8083870b1ac1c69603c"
server_key_version: "encryptionkey@0"
I20260929 18:37:41.837379 18851 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260929 18:37:41.839815 18865 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:37:41.841102 18851 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:37:41.841292 18851 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790707036041078-13712-0/minicluster-data/master-0/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790707036041078-13712-0/minicluster-data/master-0/wal
uuid: "9cfcc8219ed245d5baa67c8a06bac2bb"
format_stamp: "Formatted at 2026-09-29 18:37:41 on dist-test-slave-j1xs"
server_key: "bc4ee7ed931e5dfcd33d675ddb12e53b"
server_key_iv: "4971831d703fc8083870b1ac1c69603c"
server_key_version: "encryptionkey@0"
I20260929 18:37:41.841411 18851 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790707036041078-13712-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790707036041078-13712-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790707036041078-13712-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:37:41.859246 18851 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:37:41.859939 18851 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:37:41.860136 18851 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:37:41.867265 18851 rpc_server.cc:307] RPC server started. Bound to: 127.13.100.62:41087
I20260929 18:37:41.867275 18917 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.100.62:41087 every 8 connection(s)
I20260929 18:37:41.868180 18851 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
I20260929 18:37:41.871217 18918 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:37:41.873625 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 18851
I20260929 18:37:41.873718 13712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790707036041078-13712-0/minicluster-data/master-0/wal/instance
I20260929 18:37:41.874060 13712 external_mini_cluster.cc:1428] Setting key 9664cdc7b93477d6f9174d77f138cf11
I20260929 18:37:41.876804 18918 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9cfcc8219ed245d5baa67c8a06bac2bb: Bootstrap starting.
I20260929 18:37:41.878957 18918 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9cfcc8219ed245d5baa67c8a06bac2bb: Neither blocks nor log segments found. Creating new log.
I20260929 18:37:41.879654 18918 log.cc:824] T 00000000000000000000000000000000 P 9cfcc8219ed245d5baa67c8a06bac2bb: Log is configured to *not* fsync() on all Append() calls
I20260929 18:37:41.881359 18918 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9cfcc8219ed245d5baa67c8a06bac2bb: No bootstrap required, opened a new log
I20260929 18:37:41.882820 18921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:41.875424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46812 (local address 127.13.100.62:41087)
0929 18:37:41.876026 (+ 602us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:41.876038 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:41.876073 (+ 35us) server_negotiation.cc:407] Connection header received
0929 18:37:41.876677 (+ 604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:41.876693 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:41.877121 (+ 428us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:41.877435 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:41.877849 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:41.878827 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:41.879737 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:41.880230 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:41.880396 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:41.881203 (+ 807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:41.881208 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:41.881220 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:37:41.881625 (+ 405us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:41.881677 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:37:41.881697 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:41.881798 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:41.881949 (+ 151us) server_negotiation.cc:299] Negotiation successful
0929 18:37:41.882178 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":112,"threads_started":1}
I20260929 18:37:41.885175 18918 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9cfcc8219ed245d5baa67c8a06bac2bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cfcc8219ed245d5baa67c8a06bac2bb" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 41087 } }
I20260929 18:37:41.885394 18918 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9cfcc8219ed245d5baa67c8a06bac2bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:37:41.885455 18918 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9cfcc8219ed245d5baa67c8a06bac2bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cfcc8219ed245d5baa67c8a06bac2bb, State: Initialized, Role: FOLLOWER
I20260929 18:37:41.885571 13712 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:37:41.885874 18918 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9cfcc8219ed245d5baa67c8a06bac2bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cfcc8219ed245d5baa67c8a06bac2bb" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 41087 } }
I20260929 18:37:41.886068 18918 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9cfcc8219ed245d5baa67c8a06bac2bb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:37:41.886135 18918 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9cfcc8219ed245d5baa67c8a06bac2bb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:37:41.886231 18918 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9cfcc8219ed245d5baa67c8a06bac2bb [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:37:41.887216 18918 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9cfcc8219ed245d5baa67c8a06bac2bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cfcc8219ed245d5baa67c8a06bac2bb" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 41087 } }
I20260929 18:37:41.887526 18918 leader_election.cc:302] T 00000000000000000000000000000000 P 9cfcc8219ed245d5baa67c8a06bac2bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9cfcc8219ed245d5baa67c8a06bac2bb; no voters:
I20260929 18:37:41.887825 18918 leader_election.cc:288] T 00000000000000000000000000000000 P 9cfcc8219ed245d5baa67c8a06bac2bb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:37:41.888032 18923 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9cfcc8219ed245d5baa67c8a06bac2bb [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:37:41.888314 18923 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9cfcc8219ed245d5baa67c8a06bac2bb [term 1 LEADER]: Becoming Leader. State: Replica: 9cfcc8219ed245d5baa67c8a06bac2bb, State: Running, Role: LEADER
I20260929 18:37:41.888610 18923 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9cfcc8219ed245d5baa67c8a06bac2bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cfcc8219ed245d5baa67c8a06bac2bb" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 41087 } }
I20260929 18:37:41.889068 18918 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9cfcc8219ed245d5baa67c8a06bac2bb [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:37:41.890089 18924 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cfcc8219ed245d5baa67c8a06bac2bb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9cfcc8219ed245d5baa67c8a06bac2bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cfcc8219ed245d5baa67c8a06bac2bb" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 41087 } } }
I20260929 18:37:41.890229 18924 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cfcc8219ed245d5baa67c8a06bac2bb [sys.catalog]: This master's current role is: LEADER
I20260929 18:37:41.890720 18925 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cfcc8219ed245d5baa67c8a06bac2bb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9cfcc8219ed245d5baa67c8a06bac2bb. Latest consensus state: current_term: 1 leader_uuid: "9cfcc8219ed245d5baa67c8a06bac2bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cfcc8219ed245d5baa67c8a06bac2bb" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 41087 } } }
I20260929 18:37:41.890806 18925 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cfcc8219ed245d5baa67c8a06bac2bb [sys.catalog]: This master's current role is: LEADER
I20260929 18:37:41.890882 18932 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:37:41.893697 18932 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:37:41.899020 18932 catalog_manager.cc:1347] Generated new cluster ID: 9e010e9a0a804699a98f2b6260d56c8d
I20260929 18:37:41.899089 18932 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:37:41.908849 18932 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:37:41.909754 18932 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:37:41.917646 18932 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9cfcc8219ed245d5baa67c8a06bac2bb: Generated new TSK 0
I20260929 18:37:41.918399 18932 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260929 18:37:41.951332 18948 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260929 18:37:41.951450 18948 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260929 18:37:41.951936 18921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:41.935300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46824 (local address 127.13.100.62:41087)
0929 18:37:41.935430 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:41.935434 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:41.935651 (+ 217us) server_negotiation.cc:407] Connection header received
0929 18:37:41.935850 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:41.935853 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:41.935899 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:41.935982 (+ 83us) server_negotiation.cc:226] Negotiated authn=JWT
0929 18:37:41.936266 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:41.936796 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:41.938021 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:41.938260 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:41.938340 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:41.938742 (+ 402us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 18:37:41.951624 (+ 12882us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43375/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0929 18:37:41.951830 (+ 206us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:46824: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43375/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":37}
W20260929 18:37:41.952028 18921 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:46824: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43375/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260929 18:37:41.953341 18947 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:37:41.935171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:41087 (local address 127.0.0.1:46824)
0929 18:37:41.935541 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:41.935555 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:41.935698 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:41.936023 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:41.936026 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:41.936039 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0929 18:37:41.936160 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:41.936167 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:41.936904 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:41.936907 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:41.937878 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:41.937924 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:41.938374 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:41.938378 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:41.938468 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:41.938516 (+ 48us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 18:37:41.951827 (+ 13311us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43375/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0929 18:37:41.952064 (+ 237us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.13.100.62:41087: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43375/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":120,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.13.100.62:41087: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43375/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260929 18:37:41.954078 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 18851
2026-09-29T18:37:41Z chronyd exiting
I20260929 18:37:41.991371 13712 test_util.cc:175] -----------------------------------------------
I20260929 18:37:41.991460 13712 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790707036041078-13712-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (305 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260929 18:37:41.992547 13712 mini_oidc.cc:152] Starting JWKS server
I20260929 18:37:41.993608 13712 webserver.cc:460] Webserver started at https://127.0.0.1:43723/ using document root <none> and password file <none>
I20260929 18:37:41.993712 13712 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 18:37:41.994129 13712 webserver.cc:460] Webserver started at http://127.0.0.1:37669/ using document root <none> and password file <none>
2026-09-29T18:37:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:37:41Z Disabled control of system clock
I20260929 18:37:42.004344 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790707036041078-13712-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790707036041078-13712-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790707036041078-13712-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.100.62:39587
--webserver_interface=127.13.100.62
--webserver_port=0
--builtin_ntp_servers=127.13.100.20:45429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.100.62:39587
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43723/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790707036041078-13712-0/testchainca.cert with env {}
W20260929 18:37:42.114807 18954 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:37:42.115144 18954 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:37:42.115224 18954 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:37:42.119434 18954 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:37:42.119546 18954 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:37:42.119587 18954 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:37:42.119634 18954 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:37:42.119673 18954 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 18:37:42.119714 18954 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 18:37:42.119750 18954 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43723/jwks.json
I20260929 18:37:42.125242 18954 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.100.20:45429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790707036041078-13712-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790707036041078-13712-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.100.62:39587
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.100.62:39587
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43723/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.100.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790707036041078-13712-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18954
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790707036041078-13712-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:37:42.126415 18954 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:37:42.127590 18954 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:37:42.136057 18962 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:37:42.136057 18959 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:37:42.136119 18960 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:37:42.136335 18954 server_base.cc:1034] running on GCE node
I20260929 18:37:42.136940 18954 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:37:42.138087 18954 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:37:42.139262 18954 hybrid_clock.cc:648] HybridClock initialized: now 1790707062139244 us; error 32 us; skew 500 ppm
I20260929 18:37:42.141026 18954 webserver.cc:460] Webserver started at http://127.13.100.62:33331/ using document root <none> and password file <none>
I20260929 18:37:42.141549 18954 fs_manager.cc:359] Metadata directory not provided
I20260929 18:37:42.141616 18954 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:37:42.141773 18954 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:37:42.143827 18954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790707036041078-13712-0/minicluster-data/master-0/data/instance:
uuid: "f923019013b84e69b62635048fb6450e"
format_stamp: "Formatted at 2026-09-29 18:37:42 on dist-test-slave-j1xs"
server_key: "91d49201027fef99a0392e4e1c8d43c8"
server_key_iv: "8b06f67665e2fb037ab2a9d12dcac7bb"
server_key_version: "encryptionkey@0"
I20260929 18:37:42.144373 18954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790707036041078-13712-0/minicluster-data/master-0/wal/instance:
uuid: "f923019013b84e69b62635048fb6450e"
format_stamp: "Formatted at 2026-09-29 18:37:42 on dist-test-slave-j1xs"
server_key: "91d49201027fef99a0392e4e1c8d43c8"
server_key_iv: "8b06f67665e2fb037ab2a9d12dcac7bb"
server_key_version: "encryptionkey@0"
I20260929 18:37:42.148012 18954 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 18:37:42.150315 18968 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:37:42.151500 18954 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:37:42.151600 18954 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790707036041078-13712-0/minicluster-data/master-0/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790707036041078-13712-0/minicluster-data/master-0/wal
uuid: "f923019013b84e69b62635048fb6450e"
format_stamp: "Formatted at 2026-09-29 18:37:42 on dist-test-slave-j1xs"
server_key: "91d49201027fef99a0392e4e1c8d43c8"
server_key_iv: "8b06f67665e2fb037ab2a9d12dcac7bb"
server_key_version: "encryptionkey@0"
I20260929 18:37:42.151693 18954 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790707036041078-13712-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790707036041078-13712-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790707036041078-13712-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:37:42.166363 18954 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:37:42.166999 18954 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:37:42.167151 18954 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:37:42.174461 18954 rpc_server.cc:307] RPC server started. Bound to: 127.13.100.62:39587
I20260929 18:37:42.174553 19020 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.100.62:39587 every 8 connection(s)
I20260929 18:37:42.175413 18954 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
I20260929 18:37:42.178498 19021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:37:42.180600 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 18954
I20260929 18:37:42.180692 13712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790707036041078-13712-0/minicluster-data/master-0/wal/instance
I20260929 18:37:42.180977 13712 external_mini_cluster.cc:1428] Setting key bbfeb82b2855c5b38a13046436a769e2
I20260929 18:37:42.184314 19021 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f923019013b84e69b62635048fb6450e: Bootstrap starting.
I20260929 18:37:42.186524 19021 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f923019013b84e69b62635048fb6450e: Neither blocks nor log segments found. Creating new log.
I20260929 18:37:42.187253 19021 log.cc:824] T 00000000000000000000000000000000 P f923019013b84e69b62635048fb6450e: Log is configured to *not* fsync() on all Append() calls
I20260929 18:37:42.189590 19021 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f923019013b84e69b62635048fb6450e: No bootstrap required, opened a new log
I20260929 18:37:42.190263 19024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:42.182399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35502 (local address 127.13.100.62:39587)
0929 18:37:42.183013 (+ 614us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:42.183027 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:42.183071 (+ 44us) server_negotiation.cc:407] Connection header received
0929 18:37:42.183730 (+ 659us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:42.183746 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:42.184127 (+ 381us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:42.184452 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:42.184854 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:42.185783 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:42.186673 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:42.187138 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:42.187305 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:42.188156 (+ 851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:42.188163 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:42.188179 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:37:42.188699 (+ 520us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:42.188766 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:37:42.188806 (+ 40us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:42.188907 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:42.189053 (+ 146us) server_negotiation.cc:299] Negotiation successful
0929 18:37:42.189301 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":128,"threads_started":1}
I20260929 18:37:42.192986 13712 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:37:42.193679 19021 raft_consensus.cc:348] T 00000000000000000000000000000000 P f923019013b84e69b62635048fb6450e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f923019013b84e69b62635048fb6450e" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 39587 } }
I20260929 18:37:42.193943 19021 raft_consensus.cc:374] T 00000000000000000000000000000000 P f923019013b84e69b62635048fb6450e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:37:42.194007 19021 raft_consensus.cc:729] T 00000000000000000000000000000000 P f923019013b84e69b62635048fb6450e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f923019013b84e69b62635048fb6450e, State: Initialized, Role: FOLLOWER
I20260929 18:37:42.194460 19021 consensus_queue.cc:260] T 00000000000000000000000000000000 P f923019013b84e69b62635048fb6450e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f923019013b84e69b62635048fb6450e" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 39587 } }
I20260929 18:37:42.194645 19021 raft_consensus.cc:388] T 00000000000000000000000000000000 P f923019013b84e69b62635048fb6450e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:37:42.194729 19021 raft_consensus.cc:482] T 00000000000000000000000000000000 P f923019013b84e69b62635048fb6450e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:37:42.194837 19021 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f923019013b84e69b62635048fb6450e [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:37:42.195734 19021 raft_consensus.cc:504] T 00000000000000000000000000000000 P f923019013b84e69b62635048fb6450e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f923019013b84e69b62635048fb6450e" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 39587 } }
I20260929 18:37:42.196062 19021 leader_election.cc:302] T 00000000000000000000000000000000 P f923019013b84e69b62635048fb6450e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f923019013b84e69b62635048fb6450e; no voters:
I20260929 18:37:42.196346 19021 leader_election.cc:288] T 00000000000000000000000000000000 P f923019013b84e69b62635048fb6450e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:37:42.196502 19026 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f923019013b84e69b62635048fb6450e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:37:42.196738 19026 raft_consensus.cc:686] T 00000000000000000000000000000000 P f923019013b84e69b62635048fb6450e [term 1 LEADER]: Becoming Leader. State: Replica: f923019013b84e69b62635048fb6450e, State: Running, Role: LEADER
I20260929 18:37:42.197026 19026 consensus_queue.cc:237] T 00000000000000000000000000000000 P f923019013b84e69b62635048fb6450e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f923019013b84e69b62635048fb6450e" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 39587 } }
I20260929 18:37:42.197515 19021 sys_catalog.cc:564] T 00000000000000000000000000000000 P f923019013b84e69b62635048fb6450e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:37:42.198783 19028 sys_catalog.cc:455] T 00000000000000000000000000000000 P f923019013b84e69b62635048fb6450e [sys.catalog]: SysCatalogTable state changed. Reason: New leader f923019013b84e69b62635048fb6450e. Latest consensus state: current_term: 1 leader_uuid: "f923019013b84e69b62635048fb6450e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f923019013b84e69b62635048fb6450e" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 39587 } } }
I20260929 18:37:42.198789 19027 sys_catalog.cc:455] T 00000000000000000000000000000000 P f923019013b84e69b62635048fb6450e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f923019013b84e69b62635048fb6450e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f923019013b84e69b62635048fb6450e" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 39587 } } }
I20260929 18:37:42.198908 19028 sys_catalog.cc:458] T 00000000000000000000000000000000 P f923019013b84e69b62635048fb6450e [sys.catalog]: This master's current role is: LEADER
I20260929 18:37:42.198943 19027 sys_catalog.cc:458] T 00000000000000000000000000000000 P f923019013b84e69b62635048fb6450e [sys.catalog]: This master's current role is: LEADER
I20260929 18:37:42.199246 19035 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:37:42.202520 19035 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:37:42.207634 19035 catalog_manager.cc:1347] Generated new cluster ID: 2bcdbd76744745da8b5f93fd0482ca95
I20260929 18:37:42.207705 19035 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:37:42.215605 19035 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:37:42.216740 19035 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:37:42.232214 19035 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f923019013b84e69b62635048fb6450e: Generated new TSK 0
I20260929 18:37:42.232998 19035 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260929 18:37:42.271240 19051 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260929 18:37:42.271348 19051 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260929 18:37:42.272007 19024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:42.252033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35506 (local address 127.13.100.62:39587)
0929 18:37:42.252162 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:42.252172 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:42.252316 (+ 144us) server_negotiation.cc:407] Connection header received
0929 18:37:42.252519 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:42.252522 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:42.252568 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:42.252674 (+ 106us) server_negotiation.cc:226] Negotiated authn=JWT
0929 18:37:42.252934 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:42.253463 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:42.254696 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:42.254911 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:42.254988 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:42.255356 (+ 368us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 18:37:42.271632 (+ 16276us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43723/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0929 18:37:42.271872 (+ 240us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:35506: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43723/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":53}
W20260929 18:37:42.272090 19024 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:35506: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43723/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260929 18:37:42.272262 19050 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:37:42.251877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:39587 (local address 127.0.0.1:35506)
0929 18:37:42.252225 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:42.252240 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:42.252351 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:42.252695 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:42.252698 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:42.252706 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0929 18:37:42.252814 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:42.252820 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:42.253579 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:42.253583 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:42.254590 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:42.254600 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:42.255023 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:42.255027 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:42.255127 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:42.255155 (+ 28us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 18:37:42.271852 (+ 16697us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43723/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0929 18:37:42.272066 (+ 214us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.13.100.62:39587: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43723/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":103,"threads_started":1}
I20260929 18:37:42.272851 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 18954
2026-09-29T18:37:42Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (319 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260929 18:37:42.312286 13712 mini_oidc.cc:152] Starting JWKS server
I20260929 18:37:42.313100 13712 webserver.cc:460] Webserver started at https://127.0.0.1:38005/ using document root <none> and password file <none>
I20260929 18:37:42.313187 13712 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 18:37:42.313535 13712 webserver.cc:460] Webserver started at http://127.0.0.1:39039/ using document root <none> and password file <none>
2026-09-29T18:37:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:37:42Z Disabled control of system clock
I20260929 18:37:42.322422 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790707036041078-13712-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790707036041078-13712-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790707036041078-13712-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.100.62:43465
--webserver_interface=127.13.100.62
--webserver_port=0
--builtin_ntp_servers=127.13.100.20:35037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.100.62:43465
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38005/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260929 18:37:42.431865 19057 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:37:42.432147 19057 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:37:42.432189 19057 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:37:42.436200 19057 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:37:42.436278 19057 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:37:42.436297 19057 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:37:42.436317 19057 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:37:42.436335 19057 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 18:37:42.436354 19057 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 18:37:42.436371 19057 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38005/jwks.json
I20260929 18:37:42.441164 19057 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.100.20:35037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790707036041078-13712-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790707036041078-13712-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.100.62:43465
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.100.62:43465
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38005/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.100.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19057
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790707036041078-13712-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:37:42.442277 19057 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:37:42.443617 19057 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:37:42.452275 19063 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:37:42.452253 19065 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:37:42.452329 19062 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:37:42.452587 19057 server_base.cc:1034] running on GCE node
I20260929 18:37:42.453073 19057 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:37:42.454295 19057 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:37:42.455554 19057 hybrid_clock.cc:648] HybridClock initialized: now 1790707062455479 us; error 95 us; skew 500 ppm
I20260929 18:37:42.458101 19057 webserver.cc:460] Webserver started at http://127.13.100.62:45371/ using document root <none> and password file <none>
I20260929 18:37:42.458644 19057 fs_manager.cc:359] Metadata directory not provided
I20260929 18:37:42.458704 19057 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:37:42.458877 19057 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:37:42.460844 19057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790707036041078-13712-0/minicluster-data/master-0/data/instance:
uuid: "fa20430d1b394b469972fa5d0ad973bb"
format_stamp: "Formatted at 2026-09-29 18:37:42 on dist-test-slave-j1xs"
server_key: "06e1cc344c8f03c16f25f270e95e0648"
server_key_iv: "b11ccfd9d30c9b7da42a97780abe8eb3"
server_key_version: "encryptionkey@0"
I20260929 18:37:42.461339 19057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790707036041078-13712-0/minicluster-data/master-0/wal/instance:
uuid: "fa20430d1b394b469972fa5d0ad973bb"
format_stamp: "Formatted at 2026-09-29 18:37:42 on dist-test-slave-j1xs"
server_key: "06e1cc344c8f03c16f25f270e95e0648"
server_key_iv: "b11ccfd9d30c9b7da42a97780abe8eb3"
server_key_version: "encryptionkey@0"
I20260929 18:37:42.464960 19057 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260929 18:37:42.467324 19071 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:37:42.468581 19057 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260929 18:37:42.468685 19057 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790707036041078-13712-0/minicluster-data/master-0/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790707036041078-13712-0/minicluster-data/master-0/wal
uuid: "fa20430d1b394b469972fa5d0ad973bb"
format_stamp: "Formatted at 2026-09-29 18:37:42 on dist-test-slave-j1xs"
server_key: "06e1cc344c8f03c16f25f270e95e0648"
server_key_iv: "b11ccfd9d30c9b7da42a97780abe8eb3"
server_key_version: "encryptionkey@0"
I20260929 18:37:42.468786 19057 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790707036041078-13712-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790707036041078-13712-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790707036041078-13712-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:37:42.480400 19057 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:37:42.480969 19057 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:37:42.481113 19057 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:37:42.487877 19057 rpc_server.cc:307] RPC server started. Bound to: 127.13.100.62:43465
I20260929 18:37:42.487892 19123 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.100.62:43465 every 8 connection(s)
I20260929 18:37:42.488881 19057 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
I20260929 18:37:42.491745 19124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:37:42.496877 19124 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa20430d1b394b469972fa5d0ad973bb: Bootstrap starting.
I20260929 18:37:42.498555 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 19057
I20260929 18:37:42.498690 13712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790707036041078-13712-0/minicluster-data/master-0/wal/instance
I20260929 18:37:42.498942 13712 external_mini_cluster.cc:1428] Setting key 2ccbe61e66a529eb450fd85ac3742c62
I20260929 18:37:42.499331 19124 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fa20430d1b394b469972fa5d0ad973bb: Neither blocks nor log segments found. Creating new log.
I20260929 18:37:42.500191 19124 log.cc:824] T 00000000000000000000000000000000 P fa20430d1b394b469972fa5d0ad973bb: Log is configured to *not* fsync() on all Append() calls
I20260929 18:37:42.502152 19124 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa20430d1b394b469972fa5d0ad973bb: No bootstrap required, opened a new log
I20260929 18:37:42.505421 19124 raft_consensus.cc:348] T 00000000000000000000000000000000 P fa20430d1b394b469972fa5d0ad973bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa20430d1b394b469972fa5d0ad973bb" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 43465 } }
I20260929 18:37:42.505661 19124 raft_consensus.cc:374] T 00000000000000000000000000000000 P fa20430d1b394b469972fa5d0ad973bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:37:42.505752 19124 raft_consensus.cc:729] T 00000000000000000000000000000000 P fa20430d1b394b469972fa5d0ad973bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa20430d1b394b469972fa5d0ad973bb, State: Initialized, Role: FOLLOWER
I20260929 18:37:42.506268 19124 consensus_queue.cc:260] T 00000000000000000000000000000000 P fa20430d1b394b469972fa5d0ad973bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa20430d1b394b469972fa5d0ad973bb" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 43465 } }
I20260929 18:37:42.506434 19124 raft_consensus.cc:388] T 00000000000000000000000000000000 P fa20430d1b394b469972fa5d0ad973bb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:37:42.506503 19124 raft_consensus.cc:482] T 00000000000000000000000000000000 P fa20430d1b394b469972fa5d0ad973bb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:37:42.506608 19124 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fa20430d1b394b469972fa5d0ad973bb [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:37:42.507560 19124 raft_consensus.cc:504] T 00000000000000000000000000000000 P fa20430d1b394b469972fa5d0ad973bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa20430d1b394b469972fa5d0ad973bb" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 43465 } }
I20260929 18:37:42.507719 19127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:42.499979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41674 (local address 127.13.100.62:43465)
0929 18:37:42.500668 (+ 689us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:42.500687 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:42.500738 (+ 51us) server_negotiation.cc:407] Connection header received
0929 18:37:42.501404 (+ 666us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:42.501420 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:42.501732 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:42.502087 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:42.502493 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:42.503417 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:42.504277 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:42.504741 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:42.504900 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:42.505746 (+ 846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:42.505753 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:42.505768 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:37:42.506245 (+ 477us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:42.506297 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:37:42.506317 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:42.506427 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:42.506587 (+ 160us) server_negotiation.cc:299] Negotiation successful
0929 18:37:42.506862 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":129,"threads_started":1}
I20260929 18:37:42.507962 19124 leader_election.cc:302] T 00000000000000000000000000000000 P fa20430d1b394b469972fa5d0ad973bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fa20430d1b394b469972fa5d0ad973bb; no voters:
I20260929 18:37:42.508297 19124 leader_election.cc:288] T 00000000000000000000000000000000 P fa20430d1b394b469972fa5d0ad973bb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:37:42.508430 19129 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fa20430d1b394b469972fa5d0ad973bb [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:37:42.508669 19129 raft_consensus.cc:686] T 00000000000000000000000000000000 P fa20430d1b394b469972fa5d0ad973bb [term 1 LEADER]: Becoming Leader. State: Replica: fa20430d1b394b469972fa5d0ad973bb, State: Running, Role: LEADER
I20260929 18:37:42.508980 19129 consensus_queue.cc:237] T 00000000000000000000000000000000 P fa20430d1b394b469972fa5d0ad973bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa20430d1b394b469972fa5d0ad973bb" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 43465 } }
I20260929 18:37:42.509553 19124 sys_catalog.cc:564] T 00000000000000000000000000000000 P fa20430d1b394b469972fa5d0ad973bb [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:37:42.510528 19131 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa20430d1b394b469972fa5d0ad973bb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fa20430d1b394b469972fa5d0ad973bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa20430d1b394b469972fa5d0ad973bb" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 43465 } } }
I20260929 18:37:42.510694 19131 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa20430d1b394b469972fa5d0ad973bb [sys.catalog]: This master's current role is: LEADER
I20260929 18:37:42.510774 13712 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:37:42.512192 19130 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa20430d1b394b469972fa5d0ad973bb [sys.catalog]: SysCatalogTable state changed. Reason: New leader fa20430d1b394b469972fa5d0ad973bb. Latest consensus state: current_term: 1 leader_uuid: "fa20430d1b394b469972fa5d0ad973bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa20430d1b394b469972fa5d0ad973bb" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 43465 } } }
I20260929 18:37:42.512288 19130 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa20430d1b394b469972fa5d0ad973bb [sys.catalog]: This master's current role is: LEADER
W20260929 18:37:42.514303 19144 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fa20430d1b394b469972fa5d0ad973bb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:37:42.514386 19144 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:37:42.514461 19138 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:37:42.515096 19138 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:37:42.519840 19138 catalog_manager.cc:1347] Generated new cluster ID: 16056b98fe12458fb9e2e87ec3e66b01
I20260929 18:37:42.519899 19138 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:37:42.541574 19138 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:37:42.542821 19138 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:37:42.553449 19138 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fa20430d1b394b469972fa5d0ad973bb: Generated new TSK 0
I20260929 18:37:42.554251 19138 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:37:42.575953 19127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:42.569975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41686 (local address 127.13.100.62:43465)
0929 18:37:42.570103 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:42.570146 (+ 43us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:42.570296 (+ 150us) server_negotiation.cc:407] Connection header received
0929 18:37:42.570421 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:42.570424 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:42.570479 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:42.570607 (+ 128us) server_negotiation.cc:226] Negotiated authn=JWT
0929 18:37:42.570863 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:42.571690 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:42.572657 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:42.573013 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:42.573122 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:42.573460 (+ 338us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 18:37:42.575601 (+ 2141us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38005/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0929 18:37:42.575823 (+ 222us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:41686: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38005/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":50}
W20260929 18:37:42.576020 19127 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:41686: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38005/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260929 18:37:42.577822 19153 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:37:42.569896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:43465 (local address 127.0.0.1:41686)
0929 18:37:42.570212 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:42.570225 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:42.570318 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:42.570627 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:42.570630 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:42.570637 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0929 18:37:42.570754 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:42.570759 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:42.571809 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:42.571811 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:42.572550 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:42.572558 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:42.573153 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:42.573156 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:42.573267 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:42.573297 (+ 30us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 18:37:42.575779 (+ 2482us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38005/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0929 18:37:42.577660 (+ 1881us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.13.100.62:43465: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38005/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":229,"mutex_wait_us":1702,"thread_start_us":111,"threads_started":1}
I20260929 18:37:42.578687 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 19057
E20260929 18:37:42.586611 19154 webserver.cc:533] Webserver: sslize failed:
E20260929 18:37:42.586683 19154 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-29T18:37:42Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (304 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-29T18:37:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:37:42Z Disabled control of system clock
I20260929 18:37:42.625010 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790707036041078-13712-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790707036041078-13712-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790707036041078-13712-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.100.62:44955
--webserver_interface=127.13.100.62
--webserver_port=0
--builtin_ntp_servers=127.13.100.20:38083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.100.62:44955
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790707036041078-13712-0/testchainca.cert with env {}
W20260929 18:37:42.731577 19158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:37:42.731891 19158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:37:42.731989 19158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:37:42.736054 19158 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:37:42.736136 19158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:37:42.736166 19158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:37:42.736222 19158 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:37:42.736258 19158 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 18:37:42.736310 19158 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 18:37:42.736343 19158 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260929 18:37:42.741683 19158 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.100.20:38083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790707036041078-13712-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790707036041078-13712-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.100.62:44955
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.100.62:44955
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.100.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790707036041078-13712-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790707036041078-13712-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:37:42.742808 19158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:37:42.743824 19158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:37:42.752581 19164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:37:42.752574 19166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:37:42.752578 19163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:37:42.753118 19158 server_base.cc:1034] running on GCE node
I20260929 18:37:42.753640 19158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:37:42.754765 19158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:37:42.755959 19158 hybrid_clock.cc:648] HybridClock initialized: now 1790707062755930 us; error 42 us; skew 500 ppm
I20260929 18:37:42.757704 19158 webserver.cc:460] Webserver started at http://127.13.100.62:34893/ using document root <none> and password file <none>
I20260929 18:37:42.758327 19158 fs_manager.cc:359] Metadata directory not provided
I20260929 18:37:42.758417 19158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:37:42.758618 19158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:37:42.760855 19158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790707036041078-13712-0/minicluster-data/master-0/data/instance:
uuid: "d541fbb302ad4953bf2dbc4a3bc04d41"
format_stamp: "Formatted at 2026-09-29 18:37:42 on dist-test-slave-j1xs"
server_key: "113ecd656de2d2d15f8d2b924298ba9f"
server_key_iv: "d8f84bab0ac87746c331977f6b3dc30d"
server_key_version: "encryptionkey@0"
I20260929 18:37:42.761458 19158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790707036041078-13712-0/minicluster-data/master-0/wal/instance:
uuid: "d541fbb302ad4953bf2dbc4a3bc04d41"
format_stamp: "Formatted at 2026-09-29 18:37:42 on dist-test-slave-j1xs"
server_key: "113ecd656de2d2d15f8d2b924298ba9f"
server_key_iv: "d8f84bab0ac87746c331977f6b3dc30d"
server_key_version: "encryptionkey@0"
I20260929 18:37:42.765061 19158 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260929 18:37:42.767560 19172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:37:42.768786 19158 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 18:37:42.768941 19158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790707036041078-13712-0/minicluster-data/master-0/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790707036041078-13712-0/minicluster-data/master-0/wal
uuid: "d541fbb302ad4953bf2dbc4a3bc04d41"
format_stamp: "Formatted at 2026-09-29 18:37:42 on dist-test-slave-j1xs"
server_key: "113ecd656de2d2d15f8d2b924298ba9f"
server_key_iv: "d8f84bab0ac87746c331977f6b3dc30d"
server_key_version: "encryptionkey@0"
I20260929 18:37:42.769053 19158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790707036041078-13712-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790707036041078-13712-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790707036041078-13712-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:37:42.780877 19158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:37:42.781543 19158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:37:42.781742 19158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:37:42.789229 19158 rpc_server.cc:307] RPC server started. Bound to: 127.13.100.62:44955
I20260929 18:37:42.789233 19224 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.100.62:44955 every 8 connection(s)
I20260929 18:37:42.790280 19158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
I20260929 18:37:42.793094 19225 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:37:42.798899 19225 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d541fbb302ad4953bf2dbc4a3bc04d41: Bootstrap starting.
I20260929 18:37:42.800331 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 19158
I20260929 18:37:42.800498 13712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790707036041078-13712-0/minicluster-data/master-0/wal/instance
I20260929 18:37:42.800778 13712 external_mini_cluster.cc:1428] Setting key 3b14e74f47c8f8fb75a701b868b290b5
I20260929 18:37:42.801645 19225 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d541fbb302ad4953bf2dbc4a3bc04d41: Neither blocks nor log segments found. Creating new log.
I20260929 18:37:42.802569 19225 log.cc:824] T 00000000000000000000000000000000 P d541fbb302ad4953bf2dbc4a3bc04d41: Log is configured to *not* fsync() on all Append() calls
I20260929 18:37:42.804558 19225 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d541fbb302ad4953bf2dbc4a3bc04d41: No bootstrap required, opened a new log
I20260929 18:37:42.808137 19225 raft_consensus.cc:348] T 00000000000000000000000000000000 P d541fbb302ad4953bf2dbc4a3bc04d41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d541fbb302ad4953bf2dbc4a3bc04d41" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 44955 } }
I20260929 18:37:42.808360 19225 raft_consensus.cc:374] T 00000000000000000000000000000000 P d541fbb302ad4953bf2dbc4a3bc04d41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:37:42.808463 19225 raft_consensus.cc:729] T 00000000000000000000000000000000 P d541fbb302ad4953bf2dbc4a3bc04d41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d541fbb302ad4953bf2dbc4a3bc04d41, State: Initialized, Role: FOLLOWER
I20260929 18:37:42.808974 19225 consensus_queue.cc:260] T 00000000000000000000000000000000 P d541fbb302ad4953bf2dbc4a3bc04d41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d541fbb302ad4953bf2dbc4a3bc04d41" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 44955 } }
I20260929 18:37:42.809147 19225 raft_consensus.cc:388] T 00000000000000000000000000000000 P d541fbb302ad4953bf2dbc4a3bc04d41 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:37:42.809245 19225 raft_consensus.cc:482] T 00000000000000000000000000000000 P d541fbb302ad4953bf2dbc4a3bc04d41 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:37:42.809378 19225 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d541fbb302ad4953bf2dbc4a3bc04d41 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:37:42.809994 19228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:42.801908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35790 (local address 127.13.100.62:44955)
0929 18:37:42.802603 (+ 695us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:42.802619 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:42.802679 (+ 60us) server_negotiation.cc:407] Connection header received
0929 18:37:42.803333 (+ 654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:42.803350 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:42.803681 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:42.804011 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:42.804427 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:42.805417 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:42.806368 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:42.806841 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:42.807008 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:42.807842 (+ 834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:42.807848 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:42.807860 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:37:42.808245 (+ 385us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:42.808298 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:37:42.808317 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:42.808430 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:42.808663 (+ 233us) server_negotiation.cc:299] Negotiation successful
0929 18:37:42.809003 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":146,"threads_started":1}
I20260929 18:37:42.810662 19225 raft_consensus.cc:504] T 00000000000000000000000000000000 P d541fbb302ad4953bf2dbc4a3bc04d41 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d541fbb302ad4953bf2dbc4a3bc04d41" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 44955 } }
I20260929 18:37:42.811039 19225 leader_election.cc:302] T 00000000000000000000000000000000 P d541fbb302ad4953bf2dbc4a3bc04d41 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d541fbb302ad4953bf2dbc4a3bc04d41; no voters:
I20260929 18:37:42.812127 19225 leader_election.cc:288] T 00000000000000000000000000000000 P d541fbb302ad4953bf2dbc4a3bc04d41 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:37:42.812281 19230 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d541fbb302ad4953bf2dbc4a3bc04d41 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:37:42.812511 19230 raft_consensus.cc:686] T 00000000000000000000000000000000 P d541fbb302ad4953bf2dbc4a3bc04d41 [term 1 LEADER]: Becoming Leader. State: Replica: d541fbb302ad4953bf2dbc4a3bc04d41, State: Running, Role: LEADER
I20260929 18:37:42.812847 19230 consensus_queue.cc:237] T 00000000000000000000000000000000 P d541fbb302ad4953bf2dbc4a3bc04d41 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d541fbb302ad4953bf2dbc4a3bc04d41" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 44955 } }
I20260929 18:37:42.813349 13712 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:37:42.813553 19225 sys_catalog.cc:564] T 00000000000000000000000000000000 P d541fbb302ad4953bf2dbc4a3bc04d41 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:37:42.814725 19231 sys_catalog.cc:455] T 00000000000000000000000000000000 P d541fbb302ad4953bf2dbc4a3bc04d41 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d541fbb302ad4953bf2dbc4a3bc04d41" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d541fbb302ad4953bf2dbc4a3bc04d41" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 44955 } } }
I20260929 18:37:42.814882 19231 sys_catalog.cc:458] T 00000000000000000000000000000000 P d541fbb302ad4953bf2dbc4a3bc04d41 [sys.catalog]: This master's current role is: LEADER
I20260929 18:37:42.815281 19239 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:37:42.815599 19232 sys_catalog.cc:455] T 00000000000000000000000000000000 P d541fbb302ad4953bf2dbc4a3bc04d41 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d541fbb302ad4953bf2dbc4a3bc04d41. Latest consensus state: current_term: 1 leader_uuid: "d541fbb302ad4953bf2dbc4a3bc04d41" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d541fbb302ad4953bf2dbc4a3bc04d41" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 44955 } } }
I20260929 18:37:42.815718 19232 sys_catalog.cc:458] T 00000000000000000000000000000000 P d541fbb302ad4953bf2dbc4a3bc04d41 [sys.catalog]: This master's current role is: LEADER
I20260929 18:37:42.818907 19239 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:37:42.824829 19239 catalog_manager.cc:1347] Generated new cluster ID: 396878b82ce44b59b8782eeade012879
I20260929 18:37:42.824926 19239 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:37:42.865388 19239 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:37:42.866393 19239 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:37:42.878566 19239 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d541fbb302ad4953bf2dbc4a3bc04d41: Generated new TSK 0
I20260929 18:37:42.879277 19239 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:37:42.973040 19228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:42.887837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35804 (local address 127.13.100.62:44955)
0929 18:37:42.887954 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:42.887960 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:42.888131 (+ 171us) server_negotiation.cc:407] Connection header received
0929 18:37:42.888334 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:42.888337 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:42.888382 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:42.888493 (+ 111us) server_negotiation.cc:226] Negotiated authn=JWT
0929 18:37:42.888721 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:42.889245 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:42.890365 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:42.890595 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:42.890711 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:42.891017 (+ 306us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 18:37:42.972532 (+ 81515us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0929 18:37:42.972884 (+ 352us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:35804: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":42}
W20260929 18:37:42.973125 19228 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:35804: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260929 18:37:42.973167 19254 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:37:42.887762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:44955 (local address 127.0.0.1:35804)
0929 18:37:42.888046 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:42.888057 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:42.888181 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:42.888496 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:42.888499 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:42.888507 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0929 18:37:42.888610 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:42.888616 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:42.889395 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:42.889398 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:42.890226 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:42.890237 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:42.890724 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:42.890729 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:42.890817 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:42.890847 (+ 30us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 18:37:42.972755 (+ 81908us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0929 18:37:42.972957 (+ 202us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.13.100.62:44955: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":99,"threads_started":1}
I20260929 18:37:42.974097 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 19158
2026-09-29T18:37:42Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (376 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (119 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (121 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707036041078-13712-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:37:43 dist-test-slave-j1xs krb5kdc[19266](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:37:43 dist-test-slave-j1xs krb5kdc[19266](info): set up 2 sockets
Sep 29 18:37:43 dist-test-slave-j1xs krb5kdc[19266](info): commencing operation
krb5kdc: starting...
W20260929 18:37:44.250546 13712 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707036041078-13712-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707036041078-13712-0/krb5kdc/test-user.keytab.
Sep 29 18:37:44 dist-test-slave-j1xs krb5kdc[19266](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707064, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:37:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:37:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.100.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.100.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.100.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.62.keytab.
Entry for principal kudu/127.13.100.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.62.keytab.
I20260929 18:37:44.371528 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707036041078-13712-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707036041078-13712-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707036041078-13712-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.100.62:34665
--webserver_interface=127.13.100.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.62.keytab
--principal=kudu/127.13.100.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.100.20:37111
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.100.62:34665
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707036041078-13712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707036041078-13712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707036041078-13712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:37:44.480515 19279 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:37:44.480755 19279 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:37:44.480794 19279 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:37:44.484668 19279 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:37:44.484720 19279 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:37:44.484740 19279 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:37:44.484756 19279 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:37:44.484771 19279 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:37:44.489764 19279 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.100.20:37111
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707036041078-13712-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707036041078-13712-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.100.62:34665
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.62.keytab
--principal=kudu/127.13.100.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.100.62:34665
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.100.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19279
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707036041078-13712-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:37:44.490728 19279 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:37:44.491633 19279 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:37:44.499892 19287 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:37:44.499930 19279 server_base.cc:1034] running on GCE node
W20260929 18:37:44.499881 19285 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:37:44.499881 19284 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:37:44.500679 19279 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:37:44.501732 19279 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:37:44.502907 19279 hybrid_clock.cc:648] HybridClock initialized: now 1790707064502893 us; error 35 us; skew 500 ppm
Sep 29 18:37:44 dist-test-slave-j1xs krb5kdc[19266](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707064, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:37:44.510891 19279 init.cc:377] Logged in from keytab as kudu/127.13.100.62@KRBTEST.COM (short username kudu)
I20260929 18:37:44.511775 19279 webserver.cc:460] Webserver started at http://127.13.100.62:45729/ using document root <none> and password file <none>
I20260929 18:37:44.512358 19279 fs_manager.cc:359] Metadata directory not provided
I20260929 18:37:44.512451 19279 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:37:44.512678 19279 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:37:44.514732 19279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707036041078-13712-0/minicluster-data/master-0/data/instance:
uuid: "dce23185c015407eb196831440bc991f"
format_stamp: "Formatted at 2026-09-29 18:37:44 on dist-test-slave-j1xs"
server_key: "f29e57f07ef4bb0600bfb582d41b7f48"
server_key_iv: "1497bff41bcafb31d1416dc954446ba8"
server_key_version: "encryptionkey@0"
I20260929 18:37:44.515282 19279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707036041078-13712-0/minicluster-data/master-0/wal/instance:
uuid: "dce23185c015407eb196831440bc991f"
format_stamp: "Formatted at 2026-09-29 18:37:44 on dist-test-slave-j1xs"
server_key: "f29e57f07ef4bb0600bfb582d41b7f48"
server_key_iv: "1497bff41bcafb31d1416dc954446ba8"
server_key_version: "encryptionkey@0"
I20260929 18:37:44.518781 19279 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 18:37:44.521103 19294 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:37:44.522555 19279 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 18:37:44.522696 19279 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707036041078-13712-0/minicluster-data/master-0/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707036041078-13712-0/minicluster-data/master-0/wal
uuid: "dce23185c015407eb196831440bc991f"
format_stamp: "Formatted at 2026-09-29 18:37:44 on dist-test-slave-j1xs"
server_key: "f29e57f07ef4bb0600bfb582d41b7f48"
server_key_iv: "1497bff41bcafb31d1416dc954446ba8"
server_key_version: "encryptionkey@0"
I20260929 18:37:44.522814 19279 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707036041078-13712-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707036041078-13712-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707036041078-13712-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:37:44.562460 19279 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:37:44.565754 19279 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:37:44.565987 19279 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:37:44.572888 19279 rpc_server.cc:307] RPC server started. Bound to: 127.13.100.62:34665
I20260929 18:37:44.572916 19346 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.100.62:34665 every 8 connection(s)
I20260929 18:37:44.573796 19279 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
I20260929 18:37:44.576535 19347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:37:44.578066 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 19279
I20260929 18:37:44.578217 13712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707036041078-13712-0/minicluster-data/master-0/wal/instance
I20260929 18:37:44.578538 13712 external_mini_cluster.cc:1428] Setting key d8b47dda54de912c2a959fa8fe315562
I20260929 18:37:44.583315 19347 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dce23185c015407eb196831440bc991f: Bootstrap starting.
I20260929 18:37:44.585974 19347 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dce23185c015407eb196831440bc991f: Neither blocks nor log segments found. Creating new log.
I20260929 18:37:44.586737 19347 log.cc:824] T 00000000000000000000000000000000 P dce23185c015407eb196831440bc991f: Log is configured to *not* fsync() on all Append() calls
Sep 29 18:37:44 dist-test-slave-j1xs krb5kdc[19266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707064, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.100.62@KRBTEST.COM
I20260929 18:37:44.588887 19347 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dce23185c015407eb196831440bc991f: No bootstrap required, opened a new log
I20260929 18:37:44.593647 19347 raft_consensus.cc:348] T 00000000000000000000000000000000 P dce23185c015407eb196831440bc991f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dce23185c015407eb196831440bc991f" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 34665 } }
I20260929 18:37:44.593928 19347 raft_consensus.cc:374] T 00000000000000000000000000000000 P dce23185c015407eb196831440bc991f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:37:44.593947 19350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:44.579997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42732 (local address 127.13.100.62:34665)
0929 18:37:44.580406 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:44.580428 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:44.580470 (+ 42us) server_negotiation.cc:407] Connection header received
0929 18:37:44.581383 (+ 913us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:44.581406 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:44.581819 (+ 413us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:44.582221 (+ 402us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:44.582863 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:44.584281 (+ 1418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:44.585175 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:44.585871 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:44.586088 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:44.588781 (+ 2693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:44.588787 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:44.588803 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:37:44.588831 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:44.590782 (+ 1951us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:44.591218 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:44.591227 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:44.591241 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:44.591404 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:44.591661 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:44.591664 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:44.591666 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:44.592163 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:44.592352 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:44.592659 (+ 307us) server_negotiation.cc:299] Negotiation successful
0929 18:37:44.592972 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":102,"threads_started":1}
I20260929 18:37:44.594015 19347 raft_consensus.cc:729] T 00000000000000000000000000000000 P dce23185c015407eb196831440bc991f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dce23185c015407eb196831440bc991f, State: Initialized, Role: FOLLOWER
I20260929 18:37:44.594444 19347 consensus_queue.cc:260] T 00000000000000000000000000000000 P dce23185c015407eb196831440bc991f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dce23185c015407eb196831440bc991f" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 34665 } }
I20260929 18:37:44.594597 19347 raft_consensus.cc:388] T 00000000000000000000000000000000 P dce23185c015407eb196831440bc991f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:37:44.594664 19347 raft_consensus.cc:482] T 00000000000000000000000000000000 P dce23185c015407eb196831440bc991f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:37:44.594764 19347 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dce23185c015407eb196831440bc991f [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:37:44.595901 19347 raft_consensus.cc:504] T 00000000000000000000000000000000 P dce23185c015407eb196831440bc991f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dce23185c015407eb196831440bc991f" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 34665 } }
I20260929 18:37:44.596244 19347 leader_election.cc:302] T 00000000000000000000000000000000 P dce23185c015407eb196831440bc991f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dce23185c015407eb196831440bc991f; no voters:
I20260929 18:37:44.596562 19347 leader_election.cc:288] T 00000000000000000000000000000000 P dce23185c015407eb196831440bc991f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:37:44.596720 19352 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dce23185c015407eb196831440bc991f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:37:44.596979 19352 raft_consensus.cc:686] T 00000000000000000000000000000000 P dce23185c015407eb196831440bc991f [term 1 LEADER]: Becoming Leader. State: Replica: dce23185c015407eb196831440bc991f, State: Running, Role: LEADER
I20260929 18:37:44.597285 19352 consensus_queue.cc:237] T 00000000000000000000000000000000 P dce23185c015407eb196831440bc991f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dce23185c015407eb196831440bc991f" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 34665 } }
I20260929 18:37:44.597783 19347 sys_catalog.cc:564] T 00000000000000000000000000000000 P dce23185c015407eb196831440bc991f [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:37:44.598976 19354 sys_catalog.cc:455] T 00000000000000000000000000000000 P dce23185c015407eb196831440bc991f [sys.catalog]: SysCatalogTable state changed. Reason: New leader dce23185c015407eb196831440bc991f. Latest consensus state: current_term: 1 leader_uuid: "dce23185c015407eb196831440bc991f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dce23185c015407eb196831440bc991f" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 34665 } } }
I20260929 18:37:44.599105 19354 sys_catalog.cc:458] T 00000000000000000000000000000000 P dce23185c015407eb196831440bc991f [sys.catalog]: This master's current role is: LEADER
I20260929 18:37:44.599291 19353 sys_catalog.cc:455] T 00000000000000000000000000000000 P dce23185c015407eb196831440bc991f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dce23185c015407eb196831440bc991f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dce23185c015407eb196831440bc991f" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 34665 } } }
I20260929 18:37:44.599459 19353 sys_catalog.cc:458] T 00000000000000000000000000000000 P dce23185c015407eb196831440bc991f [sys.catalog]: This master's current role is: LEADER
I20260929 18:37:44.599579 19361 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:37:44.602686 19361 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:37:44.608346 19361 catalog_manager.cc:1347] Generated new cluster ID: 51d96c204f8d42ce90356473472b22d4
I20260929 18:37:44.608412 19361 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:37:44.614778 19361 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:37:44.615936 19361 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:37:44.622563 19361 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dce23185c015407eb196831440bc991f: Generated new TSK 0
I20260929 18:37:44.623167 19361 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.100.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.100.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.100.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.1.keytab.
Entry for principal kudu/127.13.100.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.1.keytab.
W20260929 18:37:45.491839 13712 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.13.100.1: real 0.863s user 0.000s sys 0.003s
I20260929 18:37:45.492357 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707036041078-13712-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707036041078-13712-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707036041078-13712-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707036041078-13712-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.100.1:0
--local_ip_for_outbound_sockets=127.13.100.1
--webserver_interface=127.13.100.1
--webserver_port=0
--tserver_master_addrs=127.13.100.62:34665
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.1.keytab
--principal=kudu/127.13.100.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.100.20:37111
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707036041078-13712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707036041078-13712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707036041078-13712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:37:45.605137 19373 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:37:45.605435 19373 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:37:45.605515 19373 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:37:45.609628 19373 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:37:45.609719 19373 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:37:45.609810 19373 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.100.1
I20260929 18:37:45.615082 19373 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.100.20:37111
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707036041078-13712-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707036041078-13712-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.1.keytab
--principal=kudu/127.13.100.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.100.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707036041078-13712-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.100.1
--webserver_port=0
--tserver_master_addrs=127.13.100.62:34665
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19373
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.100.1
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707036041078-13712-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:37:45.616154 19373 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:37:45.617182 19373 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:37:45.626914 19381 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:37:45.626969 19379 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:37:45.626933 19378 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:37:45.627286 19373 server_base.cc:1034] running on GCE node
I20260929 18:37:45.627696 19373 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:37:45.628250 19373 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:37:45.629411 19373 hybrid_clock.cc:648] HybridClock initialized: now 1790707065629395 us; error 28 us; skew 500 ppm
Sep 29 18:37:45 dist-test-slave-j1xs krb5kdc[19266](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707065, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:37:45.632170 19373 init.cc:377] Logged in from keytab as kudu/127.13.100.1@KRBTEST.COM (short username kudu)
I20260929 18:37:45.633036 19373 webserver.cc:460] Webserver started at http://127.13.100.1:45777/ using document root <none> and password file <none>
I20260929 18:37:45.633594 19373 fs_manager.cc:359] Metadata directory not provided
I20260929 18:37:45.633687 19373 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:37:45.633973 19373 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:37:45.636147 19373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707036041078-13712-0/minicluster-data/ts-0/data/instance:
uuid: "4525154e034b403c8be9a44c8d883155"
format_stamp: "Formatted at 2026-09-29 18:37:45 on dist-test-slave-j1xs"
server_key: "0db93baa63197fa1aa5871d723f6501f"
server_key_iv: "53312cc6c1c47e784fbd8b684e1af83f"
server_key_version: "encryptionkey@0"
I20260929 18:37:45.636722 19373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707036041078-13712-0/minicluster-data/ts-0/wal/instance:
uuid: "4525154e034b403c8be9a44c8d883155"
format_stamp: "Formatted at 2026-09-29 18:37:45 on dist-test-slave-j1xs"
server_key: "0db93baa63197fa1aa5871d723f6501f"
server_key_iv: "53312cc6c1c47e784fbd8b684e1af83f"
server_key_version: "encryptionkey@0"
I20260929 18:37:45.656963 19373 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20260929 18:37:45.659443 19388 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:37:45.660656 19373 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 18:37:45.660799 19373 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707036041078-13712-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707036041078-13712-0/minicluster-data/ts-0/wal
uuid: "4525154e034b403c8be9a44c8d883155"
format_stamp: "Formatted at 2026-09-29 18:37:45 on dist-test-slave-j1xs"
server_key: "0db93baa63197fa1aa5871d723f6501f"
server_key_iv: "53312cc6c1c47e784fbd8b684e1af83f"
server_key_version: "encryptionkey@0"
I20260929 18:37:45.660915 19373 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707036041078-13712-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707036041078-13712-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707036041078-13712-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:37:45.681932 19373 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:37:45.684837 19373 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:37:45.685052 19373 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:37:45.685642 19373 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:37:45.686479 19373 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:37:45.686553 19373 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:37:45.686618 19373 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:37:45.686666 19373 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:37:45.695824 19373 rpc_server.cc:307] RPC server started. Bound to: 127.13.100.1:37639
I20260929 18:37:45.695847 19501 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.100.1:37639 every 8 connection(s)
I20260929 18:37:45.696727 19373 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707036041078-13712-0/minicluster-data/ts-0/data/info.pb
I20260929 18:37:45.699616 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 19373
I20260929 18:37:45.699752 13712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707036041078-13712-0/minicluster-data/ts-0/wal/instance
I20260929 18:37:45.700100 13712 external_mini_cluster.cc:1428] Setting key 279311804933558b80725bfd09dc7a35
Sep 29 18:37:45 dist-test-slave-j1xs krb5kdc[19266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707065, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.1@KRBTEST.COM for kudu/127.13.100.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.13.100.2@KRBTEST.COM; defaulting to no policy
I20260929 18:37:45.712980 19504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:45.698586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.1:54977 (local address 127.13.100.62:34665)
0929 18:37:45.698875 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:45.698879 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:45.699811 (+ 932us) server_negotiation.cc:407] Connection header received
0929 18:37:45.701495 (+ 1684us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:45.701498 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:45.701551 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:45.701648 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:45.703261 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:45.704166 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:45.705484 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:45.705930 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:45.706060 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:45.709054 (+ 2994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:45.709061 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:45.709066 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:37:45.709101 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:45.711016 (+ 1915us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:45.711600 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:45.711603 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:45.711604 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:45.711680 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:45.712001 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:45.712003 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:45.712004 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:45.712180 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:45.712256 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:45.712729 (+ 473us) server_negotiation.cc:299] Negotiation successful
0929 18:37:45.712832 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":74,"threads_started":1}
I20260929 18:37:45.713846 19505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:45.699075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:34665 (local address 127.13.100.1:54977)
0929 18:37:45.699609 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:45.699683 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:45.701250 (+ 1567us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:45.701840 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:45.701849 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:45.702404 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:37:45.703091 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:45.703113 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:45.704357 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:45.704360 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:45.705326 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:45.705336 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:45.706086 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:45.706089 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:45.706321 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:45.707020 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:37:45.707039 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:37:45.708826 (+ 1787us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:37:45.711211 (+ 2385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:45.711216 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:45.711230 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:45.711500 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:45.711764 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:45.711766 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:45.711767 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:45.711917 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:45.712265 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:37:45.712270 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:37:45.712503 (+ 233us) client_negotiation.cc:769] Sending connection context
0929 18:37:45.712663 (+ 160us) client_negotiation.cc:240] Negotiation successful
0929 18:37:45.712949 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":105,"threads_started":1}
I20260929 18:37:45.715142 19502 heartbeater.cc:347] Connected to a master server at 127.13.100.62:34665
I20260929 18:37:45.715339 19502 heartbeater.cc:464] Registering TS with master...
I20260929 18:37:45.715765 19502 heartbeater.cc:511] Master 127.13.100.62:34665 requested a full tablet report, sending...
I20260929 18:37:45.717211 19311 ts_manager.cc:194] Registered new tserver with Master: 4525154e034b403c8be9a44c8d883155 (127.13.100.1:37639)
I20260929 18:37:45.718371 19311 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.100.1@KRBTEST.COM'} at 127.13.100.1:54977
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.100.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.100.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.2.keytab.
Entry for principal kudu/127.13.100.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.2.keytab.
W20260929 18:37:46.476086 13712 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.13.100.2: real 0.774s user 0.003s sys 0.000s
I20260929 18:37:46.476801 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707036041078-13712-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707036041078-13712-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707036041078-13712-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707036041078-13712-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.100.2:0
--local_ip_for_outbound_sockets=127.13.100.2
--webserver_interface=127.13.100.2
--webserver_port=0
--tserver_master_addrs=127.13.100.62:34665
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.2.keytab
--principal=kudu/127.13.100.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.100.20:37111
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707036041078-13712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707036041078-13712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707036041078-13712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:37:46.586573 19508 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:37:46.586827 19508 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:37:46.586877 19508 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:37:46.590885 19508 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:37:46.590945 19508 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:37:46.591032 19508 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.100.2
I20260929 18:37:46.596298 19508 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.100.20:37111
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707036041078-13712-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707036041078-13712-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.2.keytab
--principal=kudu/127.13.100.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.100.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707036041078-13712-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.100.2
--webserver_port=0
--tserver_master_addrs=127.13.100.62:34665
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19508
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.100.2
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707036041078-13712-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:37:46.597350 19508 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:37:46.598381 19508 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:37:46.607923 19514 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:37:46.607882 19516 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:37:46.607879 19513 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:37:46.607915 19508 server_base.cc:1034] running on GCE node
I20260929 18:37:46.608426 19508 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:37:46.609040 19508 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:37:46.610193 19508 hybrid_clock.cc:648] HybridClock initialized: now 1790707066610174 us; error 30 us; skew 500 ppm
Sep 29 18:37:46 dist-test-slave-j1xs krb5kdc[19266](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707066, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:37:46.613030 19508 init.cc:377] Logged in from keytab as kudu/127.13.100.2@KRBTEST.COM (short username kudu)
I20260929 18:37:46.613945 19508 webserver.cc:460] Webserver started at http://127.13.100.2:40081/ using document root <none> and password file <none>
I20260929 18:37:46.614490 19508 fs_manager.cc:359] Metadata directory not provided
I20260929 18:37:46.614574 19508 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:37:46.614790 19508 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:37:46.616792 19508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707036041078-13712-0/minicluster-data/ts-1/data/instance:
uuid: "a5a512d148984b578eded32a2543a4b3"
format_stamp: "Formatted at 2026-09-29 18:37:46 on dist-test-slave-j1xs"
server_key: "06da9b5d213c5ca89ff1fdd1de6b8780"
server_key_iv: "9d32c443fb2693c3d17055426fd2458a"
server_key_version: "encryptionkey@0"
I20260929 18:37:46.617350 19508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707036041078-13712-0/minicluster-data/ts-1/wal/instance:
uuid: "a5a512d148984b578eded32a2543a4b3"
format_stamp: "Formatted at 2026-09-29 18:37:46 on dist-test-slave-j1xs"
server_key: "06da9b5d213c5ca89ff1fdd1de6b8780"
server_key_iv: "9d32c443fb2693c3d17055426fd2458a"
server_key_version: "encryptionkey@0"
I20260929 18:37:46.620939 19508 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260929 18:37:46.623430 19523 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:37:46.624629 19508 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 18:37:46.624758 19508 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707036041078-13712-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707036041078-13712-0/minicluster-data/ts-1/wal
uuid: "a5a512d148984b578eded32a2543a4b3"
format_stamp: "Formatted at 2026-09-29 18:37:46 on dist-test-slave-j1xs"
server_key: "06da9b5d213c5ca89ff1fdd1de6b8780"
server_key_iv: "9d32c443fb2693c3d17055426fd2458a"
server_key_version: "encryptionkey@0"
I20260929 18:37:46.624878 19508 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707036041078-13712-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707036041078-13712-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707036041078-13712-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:37:46.638015 19508 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:37:46.640784 19508 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:37:46.640992 19508 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:37:46.641645 19508 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:37:46.642822 19508 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:37:46.642922 19508 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:37:46.643000 19508 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:37:46.643040 19508 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:37:46.655233 19508 rpc_server.cc:307] RPC server started. Bound to: 127.13.100.2:38961
I20260929 18:37:46.655252 19636 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.100.2:38961 every 8 connection(s)
I20260929 18:37:46.656169 19508 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707036041078-13712-0/minicluster-data/ts-1/data/info.pb
I20260929 18:37:46.662853 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 19508
I20260929 18:37:46.663008 13712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707036041078-13712-0/minicluster-data/ts-1/wal/instance
I20260929 18:37:46.663277 13712 external_mini_cluster.cc:1428] Setting key 2cf0b1770b167682b5dbd7fbf441adaa
Sep 29 18:37:46 dist-test-slave-j1xs krb5kdc[19266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707066, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.2@KRBTEST.COM for kudu/127.13.100.62@KRBTEST.COM
I20260929 18:37:46.670475 19639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:46.658036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.2:43269 (local address 127.13.100.62:34665)
0929 18:37:46.658299 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:46.658303 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:46.659043 (+ 740us) server_negotiation.cc:407] Connection header received
0929 18:37:46.660362 (+ 1319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:46.660365 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:46.660456 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:46.660556 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:46.661664 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:46.662273 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:46.663517 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:46.663867 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:46.663992 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:46.666663 (+ 2671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:46.666667 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:46.666669 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:37:46.666700 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:46.668296 (+ 1596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:46.668871 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:46.668875 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:46.668877 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:46.668961 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:46.669317 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:46.669320 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:46.669322 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:46.669506 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:46.669612 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:46.670163 (+ 551us) server_negotiation.cc:299] Negotiation successful
0929 18:37:46.670283 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":96,"threads_started":1}
I20260929 18:37:46.671244 19640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:46.658460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:34665 (local address 127.13.100.2:43269)
0929 18:37:46.658895 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:46.658947 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:46.660153 (+ 1206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:46.660721 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:46.660727 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:46.661071 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:37:46.661517 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:46.661530 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:46.662411 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:46.662414 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:46.663406 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:46.663416 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:46.664024 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:46.664028 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:46.664193 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:46.664809 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:37:46.664828 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:37:46.666508 (+ 1680us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:37:46.668474 (+ 1966us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:46.668480 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:46.668494 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:46.668766 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:46.669068 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:46.669072 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:46.669074 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:46.669207 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:46.669609 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:37:46.669615 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:37:46.669852 (+ 237us) client_negotiation.cc:769] Sending connection context
0929 18:37:46.670112 (+ 260us) client_negotiation.cc:240] Negotiation successful
0929 18:37:46.670428 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":111,"threads_started":1}
I20260929 18:37:46.672518 19637 heartbeater.cc:347] Connected to a master server at 127.13.100.62:34665
WARNING: no policy specified for kudu/127.13.100.3@KRBTEST.COM; defaulting to no policy
I20260929 18:37:46.672750 19637 heartbeater.cc:464] Registering TS with master...
I20260929 18:37:46.673210 19637 heartbeater.cc:511] Master 127.13.100.62:34665 requested a full tablet report, sending...
I20260929 18:37:46.674261 19311 ts_manager.cc:194] Registered new tserver with Master: a5a512d148984b578eded32a2543a4b3 (127.13.100.2:38961)
I20260929 18:37:46.674834 19311 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.100.2@KRBTEST.COM'} at 127.13.100.2:43269
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.100.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.100.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.3.keytab.
Entry for principal kudu/127.13.100.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.3.keytab.
I20260929 18:37:46.692724 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707036041078-13712-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707036041078-13712-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707036041078-13712-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707036041078-13712-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.100.3:0
--local_ip_for_outbound_sockets=127.13.100.3
--webserver_interface=127.13.100.3
--webserver_port=0
--tserver_master_addrs=127.13.100.62:34665
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.3.keytab
--principal=kudu/127.13.100.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.100.20:37111
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707036041078-13712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707036041078-13712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707036041078-13712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 18:37:46.720680 19502 heartbeater.cc:503] Master 127.13.100.62:34665 was elected leader, sending a full tablet report...
W20260929 18:37:46.805609 19643 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:37:46.805863 19643 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:37:46.806064 19643 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:37:46.810504 19643 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:37:46.810573 19643 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:37:46.810750 19643 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.100.3
I20260929 18:37:46.817086 19643 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.100.20:37111
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707036041078-13712-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707036041078-13712-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.3.keytab
--principal=kudu/127.13.100.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.100.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707036041078-13712-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.100.3
--webserver_port=0
--tserver_master_addrs=127.13.100.62:34665
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19643
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.100.3
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707036041078-13712-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:37:46.818341 19643 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:37:46.819331 19643 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:37:46.829061 19651 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:37:46.829192 19649 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:37:46.829200 19648 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:37:46.829666 19643 server_base.cc:1034] running on GCE node
I20260929 18:37:46.830504 19643 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:37:46.831176 19643 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:37:46.832389 19643 hybrid_clock.cc:648] HybridClock initialized: now 1790707066832361 us; error 55 us; skew 500 ppm
Sep 29 18:37:46 dist-test-slave-j1xs krb5kdc[19266](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707066, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:37:46.835278 19643 init.cc:377] Logged in from keytab as kudu/127.13.100.3@KRBTEST.COM (short username kudu)
I20260929 18:37:46.836163 19643 webserver.cc:460] Webserver started at http://127.13.100.3:46255/ using document root <none> and password file <none>
I20260929 18:37:46.836715 19643 fs_manager.cc:359] Metadata directory not provided
I20260929 18:37:46.836802 19643 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:37:46.837080 19643 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:37:46.839498 19643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707036041078-13712-0/minicluster-data/ts-2/data/instance:
uuid: "c7bd540278d74609962856214c4793dc"
format_stamp: "Formatted at 2026-09-29 18:37:46 on dist-test-slave-j1xs"
server_key: "bc33d1c1d76596590ebd6b4238216132"
server_key_iv: "f8e67df14e0ffea1f251f929711556c9"
server_key_version: "encryptionkey@0"
I20260929 18:37:46.840085 19643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707036041078-13712-0/minicluster-data/ts-2/wal/instance:
uuid: "c7bd540278d74609962856214c4793dc"
format_stamp: "Formatted at 2026-09-29 18:37:46 on dist-test-slave-j1xs"
server_key: "bc33d1c1d76596590ebd6b4238216132"
server_key_iv: "f8e67df14e0ffea1f251f929711556c9"
server_key_version: "encryptionkey@0"
I20260929 18:37:46.843797 19643 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 18:37:46.846307 19658 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:37:46.847553 19643 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:37:46.847699 19643 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707036041078-13712-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707036041078-13712-0/minicluster-data/ts-2/wal
uuid: "c7bd540278d74609962856214c4793dc"
format_stamp: "Formatted at 2026-09-29 18:37:46 on dist-test-slave-j1xs"
server_key: "bc33d1c1d76596590ebd6b4238216132"
server_key_iv: "f8e67df14e0ffea1f251f929711556c9"
server_key_version: "encryptionkey@0"
I20260929 18:37:46.847819 19643 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707036041078-13712-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707036041078-13712-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707036041078-13712-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:37:46.862390 19643 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:37:46.865964 19643 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:37:46.866163 19643 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:37:46.866762 19643 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:37:46.867578 19643 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:37:46.867791 19643 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:37:46.867992 19643 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:37:46.868052 19643 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:37:46.877293 19643 rpc_server.cc:307] RPC server started. Bound to: 127.13.100.3:34725
I20260929 18:37:46.877310 19771 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.100.3:34725 every 8 connection(s)
I20260929 18:37:46.878249 19643 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707036041078-13712-0/minicluster-data/ts-2/data/info.pb
I20260929 18:37:46.879284 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 19643
I20260929 18:37:46.879412 13712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707036041078-13712-0/minicluster-data/ts-2/wal/instance
I20260929 18:37:46.879701 13712 external_mini_cluster.cc:1428] Setting key 9619fbebfd4fbc7324974168120b4b18
Sep 29 18:37:46 dist-test-slave-j1xs krb5kdc[19266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707066, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.3@KRBTEST.COM for kudu/127.13.100.62@KRBTEST.COM
I20260929 18:37:46.892069 19639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:46.880173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.3:46015 (local address 127.13.100.62:34665)
0929 18:37:46.880308 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:46.880312 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:46.881321 (+ 1009us) server_negotiation.cc:407] Connection header received
0929 18:37:46.882834 (+ 1513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:46.882836 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:46.882880 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:46.882948 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:46.884053 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:46.884614 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:46.885713 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:46.886011 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:46.886096 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:46.888588 (+ 2492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:46.888591 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:46.888593 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:37:46.888617 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:46.890159 (+ 1542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:46.890655 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:46.890658 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:46.890659 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:46.890731 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:46.891021 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:46.891023 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:46.891024 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:46.891194 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:46.891267 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:46.891786 (+ 519us) server_negotiation.cc:299] Negotiation successful
0929 18:37:46.891900 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 18:37:46.892935 19774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:46.880679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:34665 (local address 127.13.100.3:46015)
0929 18:37:46.881163 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:46.881218 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:46.882644 (+ 1426us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:46.883078 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:46.883085 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:46.883436 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:37:46.883921 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:46.883935 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:46.884722 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:46.884724 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:46.885603 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:46.885613 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:46.886147 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:46.886150 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:46.886288 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:46.886851 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:37:46.886868 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:37:46.888429 (+ 1561us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:37:46.890271 (+ 1842us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:46.890277 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:46.890290 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:46.890565 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:46.890808 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:46.890810 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:46.890811 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:46.890944 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:46.891276 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:37:46.891282 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:37:46.891544 (+ 262us) client_negotiation.cc:769] Sending connection context
0929 18:37:46.891704 (+ 160us) client_negotiation.cc:240] Negotiation successful
0929 18:37:46.891963 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":112,"threads_started":1}
I20260929 18:37:46.894167 19772 heartbeater.cc:347] Connected to a master server at 127.13.100.62:34665
I20260929 18:37:46.894383 19772 heartbeater.cc:464] Registering TS with master...
I20260929 18:37:46.894824 19772 heartbeater.cc:511] Master 127.13.100.62:34665 requested a full tablet report, sending...
I20260929 18:37:46.895797 19311 ts_manager.cc:194] Registered new tserver with Master: c7bd540278d74609962856214c4793dc (127.13.100.3:34725)
I20260929 18:37:46.896412 19311 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.100.3@KRBTEST.COM'} at 127.13.100.3:46015
I20260929 18:37:46.905503 13712 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707036041078-13712-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707036041078-13712-0/krb5kdc/test-admin.keytab.
Sep 29 18:37:46 dist-test-slave-j1xs krb5kdc[19266](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707066, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:37:46.927831 13712 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260929 18:37:46.931380 19779 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:37:46.929619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50844 (local address 127.13.100.1:37639)
0929 18:37:46.930087 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:46.930096 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:46.930165 (+ 69us) server_negotiation.cc:407] Connection header received
0929 18:37:46.930374 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:46.930381 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:46.930579 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:46.930729 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:46.931199 (+ 470us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50844: BlockingRecv error: recv got EOF from 127.0.0.1:50844 (error 108)
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":110,"threads_started":1}
W20260929 18:37:46.931813 13712 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707036041078-13712-0/krb5kdc/krb5cc' of type FILE
Sep 29 18:37:46 dist-test-slave-j1xs krb5kdc[19266](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707066, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:37:46.933189 13712 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 18:37:46 dist-test-slave-j1xs krb5kdc[19266](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707066, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:37:46.934398 13712 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 18:37:46 dist-test-slave-j1xs krb5kdc[19266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707066, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.100.1@KRBTEST.COM
I20260929 18:37:46.946285 19779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:46.935390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50852 (local address 127.13.100.1:37639)
0929 18:37:46.935542 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:46.935545 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:46.935720 (+ 175us) server_negotiation.cc:407] Connection header received
0929 18:37:46.935826 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:46.935828 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:46.935867 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:46.935939 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:46.936693 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:46.937430 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:46.938885 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:46.939237 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:46.939324 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:46.942150 (+ 2826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:46.942164 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:46.942184 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:37:46.942220 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:46.944352 (+ 2132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:46.944808 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:46.944817 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:46.944821 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:46.944917 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:46.945167 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:46.945174 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:46.945177 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:46.945500 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:46.945719 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:46.945834 (+ 115us) server_negotiation.cc:299] Negotiation successful
0929 18:37:46.946016 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260929 18:37:46.950345 19779 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:37:46.949322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50864 (local address 127.13.100.1:37639)
0929 18:37:46.949460 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:46.949466 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:46.949573 (+ 107us) server_negotiation.cc:407] Connection header received
0929 18:37:46.949646 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:46.949651 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:46.949702 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:46.949793 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:46.950226 (+ 433us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50864: BlockingRecv error: recv got EOF from 127.0.0.1:50864 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260929 18:37:46.950579 13712 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707036041078-13712-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707036041078-13712-0/krb5kdc/krb5cc)
Sep 29 18:37:46 dist-test-slave-j1xs krb5kdc[19266](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707066, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:37:46.951701 13712 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 18:37:46 dist-test-slave-j1xs krb5kdc[19266](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707066, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:37:46.952682 13712 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 18:37:46 dist-test-slave-j1xs krb5kdc[19266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707066, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.100.1@KRBTEST.COM
I20260929 18:37:46.962337 19779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:46.953535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50868 (local address 127.13.100.1:37639)
0929 18:37:46.953664 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:46.953670 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:46.953713 (+ 43us) server_negotiation.cc:407] Connection header received
0929 18:37:46.953809 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:46.953814 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:46.953858 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:46.953965 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:46.954411 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:46.955330 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:46.956105 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:46.956551 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:46.956668 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:46.958926 (+ 2258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:46.958933 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:46.958938 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:37:46.958966 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:46.960696 (+ 1730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:46.961087 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:46.961092 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:46.961095 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:46.961183 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:46.961422 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:46.961426 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:46.961430 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:46.961641 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:46.961772 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:46.961986 (+ 214us) server_negotiation.cc:299] Negotiation successful
0929 18:37:46.962123 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
W20260929 18:37:46.965289 19779 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:37:46.964231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50874 (local address 127.13.100.1:37639)
0929 18:37:46.964333 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:46.964337 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:46.964473 (+ 136us) server_negotiation.cc:407] Connection header received
0929 18:37:46.964627 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:46.964630 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:46.964673 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:46.964734 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:46.965199 (+ 465us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50874: BlockingRecv error: recv got EOF from 127.0.0.1:50874 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260929 18:37:46.965406 13712 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707036041078-13712-0/krb5kdc/krb5cc' of type FILE
Sep 29 18:37:46 dist-test-slave-j1xs krb5kdc[19266](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707066, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:37:46.966487 13712 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 18:37:46 dist-test-slave-j1xs krb5kdc[19266](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707066, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:37:46.967334 13712 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 18:37:46 dist-test-slave-j1xs krb5kdc[19266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707066, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.100.1@KRBTEST.COM
I20260929 18:37:46.978694 19779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:46.968049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50882 (local address 127.13.100.1:37639)
0929 18:37:46.968151 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:46.968154 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:46.970224 (+ 2070us) server_negotiation.cc:407] Connection header received
0929 18:37:46.970336 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:46.970338 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:46.970372 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:46.970439 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:46.971120 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:46.971592 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:46.972928 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:46.973122 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:46.973194 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:46.975734 (+ 2540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:46.975738 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:46.975740 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:37:46.975760 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:46.977124 (+ 1364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:46.977538 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:46.977543 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:46.977544 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:46.977622 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:46.977920 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:46.977926 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:46.977928 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:46.978084 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:46.978214 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:46.978446 (+ 232us) server_negotiation.cc:299] Negotiation successful
0929 18:37:46.978551 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 18:37:46.979642 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 19373
I20260929 18:37:46.986079 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 19508
I20260929 18:37:46.992043 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 19643
I20260929 18:37:46.997061 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 19279
2026-09-29T18:37:46Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3783 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:37:47 dist-test-slave-j1xs krb5kdc[19792](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:37:47 dist-test-slave-j1xs krb5kdc[19792](info): set up 2 sockets
Sep 29 18:37:47 dist-test-slave-j1xs krb5kdc[19792](info): commencing operation
krb5kdc: starting...
W20260929 18:37:48.038267 13712 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/krb5kdc/test-user.keytab.
Sep 29 18:37:48 dist-test-slave-j1xs krb5kdc[19792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707068, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:37:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:37:48Z Disabled control of system clock
WARNING: no policy specified for oryx/127.13.100.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.100.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.100.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/krb5kdc/oryx_127.13.100.62.keytab.
Entry for principal oryx/127.13.100.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/krb5kdc/oryx_127.13.100.62.keytab.
I20260929 18:37:48.156325 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.100.62:34955
--webserver_interface=127.13.100.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/krb5kdc/oryx_127.13.100.62.keytab
--principal=oryx/127.13.100.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.100.20:45575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.100.62:34955
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:37:48.262708 19805 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:37:48.262959 19805 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:37:48.262998 19805 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:37:48.266961 19805 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:37:48.267014 19805 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:37:48.267033 19805 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:37:48.267050 19805 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:37:48.267102 19805 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:37:48.272238 19805 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.100.20:45575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.100.62:34955
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/krb5kdc/oryx_127.13.100.62.keytab
--principal=oryx/127.13.100.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.100.62:34955
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.100.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19805
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:37:48.273214 19805 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:37:48.274243 19805 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:37:48.282251 19810 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:37:48.282234 19813 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:37:48.282230 19811 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:37:48.282416 19805 server_base.cc:1034] running on GCE node
I20260929 18:37:48.283222 19805 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:37:48.284330 19805 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:37:48.285550 19805 hybrid_clock.cc:648] HybridClock initialized: now 1790707068285527 us; error 38 us; skew 500 ppm
Sep 29 18:37:48 dist-test-slave-j1xs krb5kdc[19792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707068, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.100.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:37:48.288391 19805 init.cc:377] Logged in from keytab as oryx/127.13.100.62@KRBTEST.COM (short username oryx)
I20260929 18:37:48.289433 19805 webserver.cc:460] Webserver started at http://127.13.100.62:45015/ using document root <none> and password file <none>
I20260929 18:37:48.290127 19805 fs_manager.cc:359] Metadata directory not provided
I20260929 18:37:48.290225 19805 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:37:48.290448 19805 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:37:48.292560 19805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/master-0/data/instance:
uuid: "16cac5b4dfe74684861c10d815fb051c"
format_stamp: "Formatted at 2026-09-29 18:37:48 on dist-test-slave-j1xs"
server_key: "aef893bf0cfa9d1d3456dbe3c7d88029"
server_key_iv: "69c072405b4c56a5aac9f6116c0a58a7"
server_key_version: "encryptionkey@0"
I20260929 18:37:48.293167 19805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/master-0/wal/instance:
uuid: "16cac5b4dfe74684861c10d815fb051c"
format_stamp: "Formatted at 2026-09-29 18:37:48 on dist-test-slave-j1xs"
server_key: "aef893bf0cfa9d1d3456dbe3c7d88029"
server_key_iv: "69c072405b4c56a5aac9f6116c0a58a7"
server_key_version: "encryptionkey@0"
I20260929 18:37:48.297045 19805 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.003s
I20260929 18:37:48.299463 19820 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:37:48.300720 19805 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 18:37:48.300887 19805 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/master-0/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/master-0/wal
uuid: "16cac5b4dfe74684861c10d815fb051c"
format_stamp: "Formatted at 2026-09-29 18:37:48 on dist-test-slave-j1xs"
server_key: "aef893bf0cfa9d1d3456dbe3c7d88029"
server_key_iv: "69c072405b4c56a5aac9f6116c0a58a7"
server_key_version: "encryptionkey@0"
I20260929 18:37:48.301010 19805 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:37:48.312258 19805 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:37:48.315352 19805 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:37:48.315569 19805 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:37:48.322777 19805 rpc_server.cc:307] RPC server started. Bound to: 127.13.100.62:34955
I20260929 18:37:48.323063 19872 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.100.62:34955 every 8 connection(s)
I20260929 18:37:48.323870 19805 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
I20260929 18:37:48.326582 19873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:37:48.332232 19873 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16cac5b4dfe74684861c10d815fb051c: Bootstrap starting.
I20260929 18:37:48.332338 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 19805
I20260929 18:37:48.332554 13712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/master-0/wal/instance
I20260929 18:37:48.332866 13712 external_mini_cluster.cc:1428] Setting key 84d2b99526d0b7371e7cf1c9edf2aa03
I20260929 18:37:48.334858 19873 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 16cac5b4dfe74684861c10d815fb051c: Neither blocks nor log segments found. Creating new log.
I20260929 18:37:48.335556 19873 log.cc:824] T 00000000000000000000000000000000 P 16cac5b4dfe74684861c10d815fb051c: Log is configured to *not* fsync() on all Append() calls
I20260929 18:37:48.337625 19873 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16cac5b4dfe74684861c10d815fb051c: No bootstrap required, opened a new log
Sep 29 18:37:48 dist-test-slave-j1xs krb5kdc[19792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707068, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.13.100.62@KRBTEST.COM
I20260929 18:37:48.342216 19873 raft_consensus.cc:348] T 00000000000000000000000000000000 P 16cac5b4dfe74684861c10d815fb051c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16cac5b4dfe74684861c10d815fb051c" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 34955 } }
I20260929 18:37:48.342473 19873 raft_consensus.cc:374] T 00000000000000000000000000000000 P 16cac5b4dfe74684861c10d815fb051c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:37:48.342547 19873 raft_consensus.cc:729] T 00000000000000000000000000000000 P 16cac5b4dfe74684861c10d815fb051c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16cac5b4dfe74684861c10d815fb051c, State: Initialized, Role: FOLLOWER
I20260929 18:37:48.342975 19873 consensus_queue.cc:260] T 00000000000000000000000000000000 P 16cac5b4dfe74684861c10d815fb051c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16cac5b4dfe74684861c10d815fb051c" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 34955 } }
I20260929 18:37:48.343120 19873 raft_consensus.cc:388] T 00000000000000000000000000000000 P 16cac5b4dfe74684861c10d815fb051c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:37:48.343205 19873 raft_consensus.cc:482] T 00000000000000000000000000000000 P 16cac5b4dfe74684861c10d815fb051c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:37:48.343333 19873 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 16cac5b4dfe74684861c10d815fb051c [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:37:48.344597 19873 raft_consensus.cc:504] T 00000000000000000000000000000000 P 16cac5b4dfe74684861c10d815fb051c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16cac5b4dfe74684861c10d815fb051c" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 34955 } }
I20260929 18:37:48.344973 19873 leader_election.cc:302] T 00000000000000000000000000000000 P 16cac5b4dfe74684861c10d815fb051c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 16cac5b4dfe74684861c10d815fb051c; no voters:
I20260929 18:37:48.345346 19873 leader_election.cc:288] T 00000000000000000000000000000000 P 16cac5b4dfe74684861c10d815fb051c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:37:48.345445 19878 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 16cac5b4dfe74684861c10d815fb051c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:37:48.345680 19878 raft_consensus.cc:686] T 00000000000000000000000000000000 P 16cac5b4dfe74684861c10d815fb051c [term 1 LEADER]: Becoming Leader. State: Replica: 16cac5b4dfe74684861c10d815fb051c, State: Running, Role: LEADER
I20260929 18:37:48.346096 19878 consensus_queue.cc:237] T 00000000000000000000000000000000 P 16cac5b4dfe74684861c10d815fb051c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16cac5b4dfe74684861c10d815fb051c" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 34955 } }
I20260929 18:37:48.346643 19873 sys_catalog.cc:564] T 00000000000000000000000000000000 P 16cac5b4dfe74684861c10d815fb051c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:37:48.347539 19879 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16cac5b4dfe74684861c10d815fb051c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 16cac5b4dfe74684861c10d815fb051c. Latest consensus state: current_term: 1 leader_uuid: "16cac5b4dfe74684861c10d815fb051c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16cac5b4dfe74684861c10d815fb051c" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 34955 } } }
I20260929 18:37:48.347657 19879 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16cac5b4dfe74684861c10d815fb051c [sys.catalog]: This master's current role is: LEADER
I20260929 18:37:48.348277 19880 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16cac5b4dfe74684861c10d815fb051c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "16cac5b4dfe74684861c10d815fb051c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16cac5b4dfe74684861c10d815fb051c" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 34955 } } }
I20260929 18:37:48.348359 19880 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16cac5b4dfe74684861c10d815fb051c [sys.catalog]: This master's current role is: LEADER
I20260929 18:37:48.348663 19876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:48.334232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51504 (local address 127.13.100.62:34955)
0929 18:37:48.334658 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:48.334673 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:48.334707 (+ 34us) server_negotiation.cc:407] Connection header received
0929 18:37:48.335396 (+ 689us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:48.335415 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:48.335781 (+ 366us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:48.336134 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:48.336931 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:48.338068 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:48.339195 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:48.339650 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:48.339765 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:48.342707 (+ 2942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:48.342712 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:48.342724 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:37:48.342751 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:48.344851 (+ 2100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:48.345331 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:48.345336 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:48.345342 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:48.345480 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:48.345926 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:48.345930 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:48.345932 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:48.346463 (+ 531us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:48.346664 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:48.347294 (+ 630us) server_negotiation.cc:299] Negotiation successful
0929 18:37:48.347626 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":138,"threads_started":1}
I20260929 18:37:48.348778 19887 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:37:48.352149 19887 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:37:48.357733 19887 catalog_manager.cc:1347] Generated new cluster ID: 3dd0e53705814f59aa70862dfbe1aea6
I20260929 18:37:48.357810 19887 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:37:48.376487 19887 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:37:48.377730 19887 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:37:48.386158 19887 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 16cac5b4dfe74684861c10d815fb051c: Generated new TSK 0
I20260929 18:37:48.387029 19887 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.13.100.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.100.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.100.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/krb5kdc/oryx_127.13.100.1.keytab.
Entry for principal oryx/127.13.100.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/krb5kdc/oryx_127.13.100.1.keytab.
I20260929 18:37:48.425463 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.100.1:0
--local_ip_for_outbound_sockets=127.13.100.1
--webserver_interface=127.13.100.1
--webserver_port=0
--tserver_master_addrs=127.13.100.62:34955
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/krb5kdc/oryx_127.13.100.1.keytab
--principal=oryx/127.13.100.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.100.20:45575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:37:48.544422 19899 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:37:48.544665 19899 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:37:48.544706 19899 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:37:48.548911 19899 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:37:48.548966 19899 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:37:48.549054 19899 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.100.1
I20260929 18:37:48.554155 19899 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.100.20:45575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/krb5kdc/oryx_127.13.100.1.keytab
--principal=oryx/127.13.100.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.100.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.100.1
--webserver_port=0
--tserver_master_addrs=127.13.100.62:34955
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19899
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.100.1
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:37:48.555199 19899 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:37:48.556182 19899 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:37:48.565707 19904 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:37:48.565732 19907 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:37:48.565871 19899 server_base.cc:1034] running on GCE node
W20260929 18:37:48.565711 19905 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:37:48.566402 19899 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:37:48.566978 19899 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:37:48.568202 19899 hybrid_clock.cc:648] HybridClock initialized: now 1790707068568177 us; error 43 us; skew 500 ppm
Sep 29 18:37:48 dist-test-slave-j1xs krb5kdc[19792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707068, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.100.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:37:48.571074 19899 init.cc:377] Logged in from keytab as oryx/127.13.100.1@KRBTEST.COM (short username oryx)
I20260929 18:37:48.571857 19899 webserver.cc:460] Webserver started at http://127.13.100.1:46753/ using document root <none> and password file <none>
I20260929 18:37:48.572351 19899 fs_manager.cc:359] Metadata directory not provided
I20260929 18:37:48.572407 19899 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:37:48.572577 19899 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:37:48.574546 19899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/ts-0/data/instance:
uuid: "8600dd95d7d441f9aaf8fc399cfccb14"
format_stamp: "Formatted at 2026-09-29 18:37:48 on dist-test-slave-j1xs"
server_key: "b7611c10c7dc7613975cee612df6b171"
server_key_iv: "59383d66760491e868fd84a5dc68ece3"
server_key_version: "encryptionkey@0"
I20260929 18:37:48.575023 19899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/ts-0/wal/instance:
uuid: "8600dd95d7d441f9aaf8fc399cfccb14"
format_stamp: "Formatted at 2026-09-29 18:37:48 on dist-test-slave-j1xs"
server_key: "b7611c10c7dc7613975cee612df6b171"
server_key_iv: "59383d66760491e868fd84a5dc68ece3"
server_key_version: "encryptionkey@0"
I20260929 18:37:48.578585 19899 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 18:37:48.581060 19914 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:37:48.582319 19899 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260929 18:37:48.582438 19899 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/ts-0/wal
uuid: "8600dd95d7d441f9aaf8fc399cfccb14"
format_stamp: "Formatted at 2026-09-29 18:37:48 on dist-test-slave-j1xs"
server_key: "b7611c10c7dc7613975cee612df6b171"
server_key_iv: "59383d66760491e868fd84a5dc68ece3"
server_key_version: "encryptionkey@0"
I20260929 18:37:48.582566 19899 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:37:48.599224 19899 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:37:48.602645 19899 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:37:48.602880 19899 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:37:48.603494 19899 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:37:48.604410 19899 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:37:48.604463 19899 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:37:48.604542 19899 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:37:48.604581 19899 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:37:48.614223 19899 rpc_server.cc:307] RPC server started. Bound to: 127.13.100.1:38831
I20260929 18:37:48.614274 20027 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.100.1:38831 every 8 connection(s)
I20260929 18:37:48.615217 19899 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/ts-0/data/info.pb
I20260929 18:37:48.623162 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 19899
I20260929 18:37:48.623311 13712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/ts-0/wal/instance
I20260929 18:37:48.623684 13712 external_mini_cluster.cc:1428] Setting key 9d4b363aedf65c39bd76c44b07dc9b5b
Sep 29 18:37:48 dist-test-slave-j1xs krb5kdc[19792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707068, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.100.1@KRBTEST.COM for oryx/127.13.100.62@KRBTEST.COM
I20260929 18:37:48.629432 19876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:48.617055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.1:47043 (local address 127.13.100.62:34955)
0929 18:37:48.617192 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:48.617195 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:48.618063 (+ 868us) server_negotiation.cc:407] Connection header received
0929 18:37:48.619390 (+ 1327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:48.619393 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:48.619451 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:48.619559 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:48.620688 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:48.621209 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:48.622385 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:48.622620 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:48.622733 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:48.625430 (+ 2697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:48.625432 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:48.625434 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:37:48.625456 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:48.626926 (+ 1470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:48.627534 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:48.627536 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:48.627537 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:48.627609 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:48.628053 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:48.628059 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:48.628063 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:48.628310 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:48.628416 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:48.629028 (+ 612us) server_negotiation.cc:299] Negotiation successful
0929 18:37:48.629184 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 18:37:48.630234 20030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:48.617440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:34955 (local address 127.13.100.1:47043)
0929 18:37:48.617858 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:48.617949 (+ 91us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:48.619182 (+ 1233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:48.619739 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:48.619746 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:48.620084 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:37:48.620534 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:48.620547 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:48.621335 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:48.621339 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:48.622263 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:48.622273 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:48.622772 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:48.622776 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:48.622956 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:48.623599 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:37:48.623623 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:37:48.625256 (+ 1633us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:37:48.627040 (+ 1784us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:48.627047 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:48.627063 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:48.627438 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:48.627699 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:48.627703 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:48.627707 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:48.627905 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:48.628450 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:37:48.628455 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:37:48.628675 (+ 220us) client_negotiation.cc:769] Sending connection context
0929 18:37:48.628964 (+ 289us) client_negotiation.cc:240] Negotiation successful
0929 18:37:48.629254 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":102,"threads_started":1}
I20260929 18:37:48.631615 20028 heartbeater.cc:347] Connected to a master server at 127.13.100.62:34955
I20260929 18:37:48.631868 20028 heartbeater.cc:464] Registering TS with master...
I20260929 18:37:48.632465 20028 heartbeater.cc:511] Master 127.13.100.62:34955 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.13.100.2@KRBTEST.COM; defaulting to no policy
I20260929 18:37:48.634152 19837 ts_manager.cc:194] Registered new tserver with Master: 8600dd95d7d441f9aaf8fc399cfccb14 (127.13.100.1:38831)
I20260929 18:37:48.635268 19837 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.13.100.1@KRBTEST.COM'} at 127.13.100.1:47043
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.100.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.100.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/krb5kdc/oryx_127.13.100.2.keytab.
Entry for principal oryx/127.13.100.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/krb5kdc/oryx_127.13.100.2.keytab.
I20260929 18:37:48.652578 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.100.2:0
--local_ip_for_outbound_sockets=127.13.100.2
--webserver_interface=127.13.100.2
--webserver_port=0
--tserver_master_addrs=127.13.100.62:34955
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/krb5kdc/oryx_127.13.100.2.keytab
--principal=oryx/127.13.100.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.100.20:45575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:37:48.759629 20033 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:37:48.759920 20033 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:37:48.759990 20033 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:37:48.764038 20033 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:37:48.764114 20033 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:37:48.764204 20033 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.100.2
I20260929 18:37:48.769241 20033 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.100.20:45575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/krb5kdc/oryx_127.13.100.2.keytab
--principal=oryx/127.13.100.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.100.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.100.2
--webserver_port=0
--tserver_master_addrs=127.13.100.62:34955
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20033
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.100.2
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:37:48.770270 20033 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:37:48.771239 20033 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:37:48.780470 20038 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:37:48.780505 20039 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:37:48.780510 20041 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:37:48.780725 20033 server_base.cc:1034] running on GCE node
I20260929 18:37:48.781213 20033 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:37:48.781860 20033 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:37:48.783067 20033 hybrid_clock.cc:648] HybridClock initialized: now 1790707068783038 us; error 42 us; skew 500 ppm
Sep 29 18:37:48 dist-test-slave-j1xs krb5kdc[19792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707068, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.100.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:37:48.785804 20033 init.cc:377] Logged in from keytab as oryx/127.13.100.2@KRBTEST.COM (short username oryx)
I20260929 18:37:48.786736 20033 webserver.cc:460] Webserver started at http://127.13.100.2:32899/ using document root <none> and password file <none>
I20260929 18:37:48.787281 20033 fs_manager.cc:359] Metadata directory not provided
I20260929 18:37:48.787338 20033 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:37:48.787564 20033 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:37:48.789582 20033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/ts-1/data/instance:
uuid: "a579500ca64f489886008b02eba510bf"
format_stamp: "Formatted at 2026-09-29 18:37:48 on dist-test-slave-j1xs"
server_key: "7dca658eda06cb9ea771ca699bc6c1fd"
server_key_iv: "6722dcfdfae663ac7e9c6c40b3bb39f1"
server_key_version: "encryptionkey@0"
I20260929 18:37:48.790158 20033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/ts-1/wal/instance:
uuid: "a579500ca64f489886008b02eba510bf"
format_stamp: "Formatted at 2026-09-29 18:37:48 on dist-test-slave-j1xs"
server_key: "7dca658eda06cb9ea771ca699bc6c1fd"
server_key_iv: "6722dcfdfae663ac7e9c6c40b3bb39f1"
server_key_version: "encryptionkey@0"
I20260929 18:37:48.793704 20033 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260929 18:37:48.796177 20048 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:37:48.797420 20033 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:37:48.797572 20033 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/ts-1/wal
uuid: "a579500ca64f489886008b02eba510bf"
format_stamp: "Formatted at 2026-09-29 18:37:48 on dist-test-slave-j1xs"
server_key: "7dca658eda06cb9ea771ca699bc6c1fd"
server_key_iv: "6722dcfdfae663ac7e9c6c40b3bb39f1"
server_key_version: "encryptionkey@0"
I20260929 18:37:48.797698 20033 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:37:48.812395 20033 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:37:48.815548 20033 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:37:48.815745 20033 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:37:48.816358 20033 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:37:48.817152 20033 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:37:48.817217 20033 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:37:48.817282 20033 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:37:48.817328 20033 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:37:48.826109 20033 rpc_server.cc:307] RPC server started. Bound to: 127.13.100.2:45533
I20260929 18:37:48.826138 20161 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.100.2:45533 every 8 connection(s)
I20260929 18:37:48.826942 20033 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/ts-1/data/info.pb
I20260929 18:37:48.828612 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 20033
I20260929 18:37:48.828734 13712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/ts-1/wal/instance
I20260929 18:37:48.829032 13712 external_mini_cluster.cc:1428] Setting key 57e04fa4f02ce1b48d5be043b1ecebd7
Sep 29 18:37:48 dist-test-slave-j1xs krb5kdc[19792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707068, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.100.2@KRBTEST.COM for oryx/127.13.100.62@KRBTEST.COM
WARNING: no policy specified for oryx/127.13.100.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.100.3@KRBTEST.COM" created.
I20260929 18:37:48.841864 19876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:48.828684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.2:39355 (local address 127.13.100.62:34955)
0929 18:37:48.828832 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:48.828836 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:48.829746 (+ 910us) server_negotiation.cc:407] Connection header received
0929 18:37:48.831147 (+ 1401us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:48.831153 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:48.831213 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:48.831306 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:48.832671 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:48.833177 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:48.834977 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:48.835186 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:48.835267 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:48.838345 (+ 3078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:48.838348 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:48.838350 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:37:48.838374 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:48.839860 (+ 1486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:48.840353 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:48.840356 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:48.840358 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:48.840437 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:48.840771 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:48.840774 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:48.840776 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:48.840951 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:48.841057 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:48.841582 (+ 525us) server_negotiation.cc:299] Negotiation successful
0929 18:37:48.841693 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 18:37:48.842694 20164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:48.829121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:34955 (local address 127.13.100.2:39355)
0929 18:37:48.829602 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:48.829652 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:48.830932 (+ 1280us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:48.831450 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:48.831456 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:48.831824 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:37:48.832521 (+ 697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:48.832540 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:48.833292 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:48.833298 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:48.834843 (+ 1545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:48.834854 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:48.835347 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:48.835353 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:48.835619 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:48.836246 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:37:48.836272 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:37:48.838156 (+ 1884us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:37:48.839977 (+ 1821us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:48.839983 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:48.839997 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:48.840246 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:48.840534 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:48.840537 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:48.840540 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:48.840669 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:48.841061 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:37:48.841069 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:37:48.841296 (+ 227us) client_negotiation.cc:769] Sending connection context
0929 18:37:48.841516 (+ 220us) client_negotiation.cc:240] Negotiation successful
0929 18:37:48.841825 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":132,"threads_started":1}
I20260929 18:37:48.843820 20162 heartbeater.cc:347] Connected to a master server at 127.13.100.62:34955
I20260929 18:37:48.844096 20162 heartbeater.cc:464] Registering TS with master...
I20260929 18:37:48.844611 20162 heartbeater.cc:511] Master 127.13.100.62:34955 requested a full tablet report, sending...
I20260929 18:37:48.845752 19837 ts_manager.cc:194] Registered new tserver with Master: a579500ca64f489886008b02eba510bf (127.13.100.2:45533)
I20260929 18:37:48.846630 19837 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.13.100.2@KRBTEST.COM'} at 127.13.100.2:39355
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.100.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/krb5kdc/oryx_127.13.100.3.keytab.
Entry for principal oryx/127.13.100.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/krb5kdc/oryx_127.13.100.3.keytab.
I20260929 18:37:48.856731 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.100.3:0
--local_ip_for_outbound_sockets=127.13.100.3
--webserver_interface=127.13.100.3
--webserver_port=0
--tserver_master_addrs=127.13.100.62:34955
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/krb5kdc/oryx_127.13.100.3.keytab
--principal=oryx/127.13.100.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.100.20:45575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:37:48.980185 20167 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:37:48.980463 20167 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:37:48.980572 20167 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:37:48.984907 20167 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:37:48.984978 20167 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:37:48.985070 20167 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.100.3
I20260929 18:37:48.990810 20167 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.100.20:45575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/krb5kdc/oryx_127.13.100.3.keytab
--principal=oryx/127.13.100.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.100.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.100.3
--webserver_port=0
--tserver_master_addrs=127.13.100.62:34955
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20167
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.100.3
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:37:48.991876 20167 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:37:48.992890 20167 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:37:49.002354 20173 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:37:49.002424 20175 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:37:49.002424 20172 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:37:49.002774 20167 server_base.cc:1034] running on GCE node
I20260929 18:37:49.003185 20167 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:37:49.003829 20167 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:37:49.004998 20167 hybrid_clock.cc:648] HybridClock initialized: now 1790707069004977 us; error 26 us; skew 500 ppm
Sep 29 18:37:49 dist-test-slave-j1xs krb5kdc[19792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707069, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.100.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:37:49.007951 20167 init.cc:377] Logged in from keytab as oryx/127.13.100.3@KRBTEST.COM (short username oryx)
I20260929 18:37:49.008774 20167 webserver.cc:460] Webserver started at http://127.13.100.3:34909/ using document root <none> and password file <none>
I20260929 18:37:49.009295 20167 fs_manager.cc:359] Metadata directory not provided
I20260929 18:37:49.009362 20167 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:37:49.009527 20167 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:37:49.011583 20167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/ts-2/data/instance:
uuid: "2fbaf12215ba453bb80f9cccaaff2e73"
format_stamp: "Formatted at 2026-09-29 18:37:49 on dist-test-slave-j1xs"
server_key: "b10c7927b6ad724fcd455be21df63d8f"
server_key_iv: "737a4e85ca88414ece4a1c49e1cbc3af"
server_key_version: "encryptionkey@0"
I20260929 18:37:49.012095 20167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/ts-2/wal/instance:
uuid: "2fbaf12215ba453bb80f9cccaaff2e73"
format_stamp: "Formatted at 2026-09-29 18:37:49 on dist-test-slave-j1xs"
server_key: "b10c7927b6ad724fcd455be21df63d8f"
server_key_iv: "737a4e85ca88414ece4a1c49e1cbc3af"
server_key_version: "encryptionkey@0"
I20260929 18:37:49.016100 20167 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260929 18:37:49.018719 20182 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:37:49.019994 20167 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:37:49.020145 20167 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/ts-2/wal
uuid: "2fbaf12215ba453bb80f9cccaaff2e73"
format_stamp: "Formatted at 2026-09-29 18:37:49 on dist-test-slave-j1xs"
server_key: "b10c7927b6ad724fcd455be21df63d8f"
server_key_iv: "737a4e85ca88414ece4a1c49e1cbc3af"
server_key_version: "encryptionkey@0"
I20260929 18:37:49.020264 20167 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:37:49.044433 20167 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:37:49.047839 20167 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:37:49.048039 20167 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:37:49.048661 20167 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:37:49.049477 20167 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:37:49.049547 20167 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:37:49.049618 20167 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:37:49.049669 20167 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:37:49.058578 20167 rpc_server.cc:307] RPC server started. Bound to: 127.13.100.3:35631
I20260929 18:37:49.058599 20295 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.100.3:35631 every 8 connection(s)
I20260929 18:37:49.059442 20167 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/ts-2/data/info.pb
I20260929 18:37:49.063548 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 20167
I20260929 18:37:49.063700 13712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/ts-2/wal/instance
I20260929 18:37:49.064047 13712 external_mini_cluster.cc:1428] Setting key 9b26530d9c875865e76f71c837dc17a5
Sep 29 18:37:49 dist-test-slave-j1xs krb5kdc[19792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707069, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.100.3@KRBTEST.COM for oryx/127.13.100.62@KRBTEST.COM
I20260929 18:37:49.073613 19876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:49.061210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.3:37799 (local address 127.13.100.62:34955)
0929 18:37:49.061370 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:49.061375 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:49.062175 (+ 800us) server_negotiation.cc:407] Connection header received
0929 18:37:49.063682 (+ 1507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:49.063687 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:49.063751 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:49.063822 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:49.065601 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:49.066390 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:49.067521 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:49.067727 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:49.067815 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:49.070238 (+ 2423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:49.070241 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:49.070243 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:37:49.070266 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:49.071736 (+ 1470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:49.072240 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:49.072243 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:49.072245 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:49.072324 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:49.072662 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:49.072664 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:49.072666 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:49.072843 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:49.072925 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:49.073375 (+ 450us) server_negotiation.cc:299] Negotiation successful
0929 18:37:49.073492 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 18:37:49.074401 20298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:49.061538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:34955 (local address 127.13.100.3:37799)
0929 18:37:49.062032 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:49.062076 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:49.063365 (+ 1289us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:49.064249 (+ 884us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:49.064258 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:49.064733 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:37:49.065437 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:49.065459 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:49.066521 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:49.066524 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:49.067382 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:49.067390 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:49.067878 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:49.067881 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:49.068029 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:49.068599 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:37:49.068614 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:37:49.070084 (+ 1470us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:37:49.071871 (+ 1787us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:49.071876 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:49.071886 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:49.072130 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:49.072411 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:49.072413 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:49.072414 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:49.072559 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:49.072934 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:37:49.072939 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:37:49.073145 (+ 206us) client_negotiation.cc:769] Sending connection context
0929 18:37:49.073291 (+ 146us) client_negotiation.cc:240] Negotiation successful
0929 18:37:49.073540 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":96,"threads_started":1}
I20260929 18:37:49.075639 20296 heartbeater.cc:347] Connected to a master server at 127.13.100.62:34955
I20260929 18:37:49.075935 20296 heartbeater.cc:464] Registering TS with master...
I20260929 18:37:49.076421 20296 heartbeater.cc:511] Master 127.13.100.62:34955 requested a full tablet report, sending...
I20260929 18:37:49.077613 19837 ts_manager.cc:194] Registered new tserver with Master: 2fbaf12215ba453bb80f9cccaaff2e73 (127.13.100.3:35631)
I20260929 18:37:49.078260 13712 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 18:37:49.078372 19837 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.13.100.3@KRBTEST.COM'} at 127.13.100.3:37799
WARNING: no policy specified for oryx/127.13.100.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.100.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.100.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/krb5kdc/oryx_127.13.100.61.keytab.
Entry for principal oryx/127.13.100.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/krb5kdc/oryx_127.13.100.61.keytab.
I20260929 18:37:49.105521 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.100.61:46389
--webserver_interface=127.13.100.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/krb5kdc/oryx_127.13.100.61.keytab
--principal=oryx/127.13.100.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.100.20:45575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.100.62:34955,127.13.100.61:46389
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:37:49.216144 20301 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:37:49.216383 20301 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:37:49.216423 20301 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:37:49.220474 20301 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:37:49.220525 20301 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:37:49.220551 20301 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:37:49.220567 20301 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:37:49.220615 20301 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:37:49.225800 20301 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.100.20:45575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.13.100.62:34955,127.13.100.61:46389
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/krb5kdc/oryx_127.13.100.61.keytab
--principal=oryx/127.13.100.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.100.61:46389
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.13.100.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20301
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
W20260929 18:37:49.226018 20301 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.13.100.62:34955,127.13.100.61:46389'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260929 18:37:49.226845 20301 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:37:49.227813 20301 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:37:49.235997 20309 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:37:49.236068 20306 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:37:49.236155 20301 server_base.cc:1034] running on GCE node
W20260929 18:37:49.236161 20307 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:37:49.236768 20301 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:37:49.237756 20301 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:37:49.238942 20301 hybrid_clock.cc:648] HybridClock initialized: now 1790707069238922 us; error 39 us; skew 500 ppm
Sep 29 18:37:49 dist-test-slave-j1xs krb5kdc[19792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707069, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.100.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:37:49.241472 20301 init.cc:377] Logged in from keytab as oryx/127.13.100.61@KRBTEST.COM (short username oryx)
I20260929 18:37:49.242336 20301 webserver.cc:460] Webserver started at http://127.13.100.61:41157/ using document root <none> and password file <none>
I20260929 18:37:49.242856 20301 fs_manager.cc:359] Metadata directory not provided
I20260929 18:37:49.242944 20301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:37:49.243162 20301 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:37:49.245074 20301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/master-1/data/instance:
uuid: "30e554990e1d47fb8e2b183647f95d5d"
format_stamp: "Formatted at 2026-09-29 18:37:49 on dist-test-slave-j1xs"
server_key: "1e18e38e77941864ee75094114de4ee6"
server_key_iv: "22b782de56a8e10975aba54c04842d36"
server_key_version: "encryptionkey@0"
I20260929 18:37:49.245637 20301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/master-1/wal/instance:
uuid: "30e554990e1d47fb8e2b183647f95d5d"
format_stamp: "Formatted at 2026-09-29 18:37:49 on dist-test-slave-j1xs"
server_key: "1e18e38e77941864ee75094114de4ee6"
server_key_iv: "22b782de56a8e10975aba54c04842d36"
server_key_version: "encryptionkey@0"
I20260929 18:37:49.249542 20301 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260929 18:37:49.251848 20316 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:37:49.253015 20301 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:37:49.253145 20301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/master-1/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/master-1/wal
uuid: "30e554990e1d47fb8e2b183647f95d5d"
format_stamp: "Formatted at 2026-09-29 18:37:49 on dist-test-slave-j1xs"
server_key: "1e18e38e77941864ee75094114de4ee6"
server_key_iv: "22b782de56a8e10975aba54c04842d36"
server_key_version: "encryptionkey@0"
I20260929 18:37:49.253247 20301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:37:49.263108 20301 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:37:49.265738 20301 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:37:49.265983 20301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:37:49.272323 20301 rpc_server.cc:307] RPC server started. Bound to: 127.13.100.61:46389
I20260929 18:37:49.272621 20368 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.100.61:46389 every 8 connection(s)
I20260929 18:37:49.273232 20301 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/master-1/data/info.pb
I20260929 18:37:49.275969 20369 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:37:49.279537 20369 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.100.62" port: 34955 } has no permanent_uuid. Determining permanent_uuid...
I20260929 18:37:49.281153 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 20301
Sep 29 18:37:49 dist-test-slave-j1xs krb5kdc[19792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707069, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.100.61@KRBTEST.COM for oryx/127.13.100.62@KRBTEST.COM
Sep 29 18:37:49 dist-test-slave-j1xs krb5kdc[19792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707068, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.13.100.61@KRBTEST.COM
I20260929 18:37:49.294485 19876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:49.281424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51510 (local address 127.13.100.62:34955)
0929 18:37:49.281582 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:49.281587 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:49.282608 (+ 1021us) server_negotiation.cc:407] Connection header received
0929 18:37:49.283475 (+ 867us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:49.283478 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:49.283527 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:49.283625 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:49.284798 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:49.285619 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:49.287180 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:49.287398 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:49.287511 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:49.290276 (+ 2765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:49.290279 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:49.290282 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:37:49.290306 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:49.292016 (+ 1710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:49.292671 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:49.292674 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:49.292675 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:49.292763 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:49.293190 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:49.293194 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:49.293195 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:49.293445 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:49.293570 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:49.294139 (+ 569us) server_negotiation.cc:299] Negotiation successful
0929 18:37:49.294272 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 18:37:49.295325 20373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:49.282127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48266 (local address 127.13.100.61:46389)
0929 18:37:49.282611 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:49.282623 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:49.282667 (+ 44us) server_negotiation.cc:407] Connection header received
0929 18:37:49.283263 (+ 596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:49.283279 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:49.283510 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:49.283703 (+ 193us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:49.284650 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:49.285985 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:49.287127 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:49.287505 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:49.287741 (+ 236us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:49.290682 (+ 2941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:49.290688 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:49.290698 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:37:49.290717 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:49.292723 (+ 2006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:49.293159 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:49.293166 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:49.293167 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:49.293247 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:49.293530 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:49.293541 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:49.293543 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:49.293824 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:49.294054 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:49.294283 (+ 229us) server_negotiation.cc:299] Negotiation successful
0929 18:37:49.294541 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":122,"server-negotiator.queue_time_us":365,"spinlock_wait_cycles":768,"thread_start_us":148,"threads_started":1}
I20260929 18:37:49.295334 20372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:49.281952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:34955 (local address 127.0.0.1:51510)
0929 18:37:49.282435 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:49.282498 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:49.283280 (+ 782us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:49.283746 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:49.283752 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:49.284243 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:37:49.284658 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:49.284672 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:49.285771 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:49.285774 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:49.286980 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:49.286990 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:49.287585 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:49.287591 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:49.287786 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:49.288513 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:37:49.288529 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:37:49.290080 (+ 1551us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:37:49.292139 (+ 2059us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:49.292146 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:49.292159 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:49.292526 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:49.292904 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:49.292908 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:49.292910 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:49.293071 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:49.293566 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:37:49.293573 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:37:49.293838 (+ 265us) client_negotiation.cc:769] Sending connection context
0929 18:37:49.294082 (+ 244us) client_negotiation.cc:240] Negotiation successful
0929 18:37:49.294467 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"mutex_wait_us":55,"thread_start_us":114,"threads_started":1}
I20260929 18:37:49.296782 20369 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.100.61" port: 46389 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 18:37:49 dist-test-slave-j1xs krb5kdc[19792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707069, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.100.61@KRBTEST.COM for oryx/127.13.100.61@KRBTEST.COM
I20260929 18:37:49.304342 20372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:49.297304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.61:46389 (local address 127.0.0.1:48278)
0929 18:37:49.297403 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:49.297416 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:49.297490 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:49.297724 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:49.297727 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:49.297879 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:37:49.298016 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:49.298021 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:49.298713 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:49.298717 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:49.299215 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:49.299222 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:49.299650 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:49.299653 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:49.299739 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:49.300183 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:37:49.300199 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:37:49.301404 (+ 1205us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:37:49.302962 (+ 1558us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:49.302965 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:49.302967 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:49.303145 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:49.303402 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:49.303405 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:49.303407 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:49.303490 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:49.303831 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:37:49.303834 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:37:49.303910 (+ 76us) client_negotiation.cc:769] Sending connection context
0929 18:37:49.304020 (+ 110us) client_negotiation.cc:240] Negotiation successful
0929 18:37:49.304148 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260929 18:37:49.304346 20373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:49.297348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48278 (local address 127.13.100.61:46389)
0929 18:37:49.297473 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:49.297477 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:49.297489 (+ 12us) server_negotiation.cc:407] Connection header received
0929 18:37:49.297592 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:49.297597 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:49.297639 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:49.297717 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:49.298113 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:49.298607 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:49.299322 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:49.299552 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:49.299656 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:49.301522 (+ 1866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:49.301525 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:49.301528 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:37:49.301556 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:49.302851 (+ 1295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:49.303236 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:49.303242 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:49.303244 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:49.303311 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:49.303584 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:49.303587 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:49.303589 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:49.303737 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:49.303832 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:49.304070 (+ 238us) server_negotiation.cc:299] Negotiation successful
0929 18:37:49.304200 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":35,"server-negotiator.queue_time_us":38}
I20260929 18:37:49.307799 20369 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30e554990e1d47fb8e2b183647f95d5d: Bootstrap starting.
I20260929 18:37:49.309996 20369 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 30e554990e1d47fb8e2b183647f95d5d: Neither blocks nor log segments found. Creating new log.
I20260929 18:37:49.310729 20369 log.cc:824] T 00000000000000000000000000000000 P 30e554990e1d47fb8e2b183647f95d5d: Log is configured to *not* fsync() on all Append() calls
I20260929 18:37:49.312446 20369 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30e554990e1d47fb8e2b183647f95d5d: No bootstrap required, opened a new log
I20260929 18:37:49.315505 20369 raft_consensus.cc:348] T 00000000000000000000000000000000 P 30e554990e1d47fb8e2b183647f95d5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16cac5b4dfe74684861c10d815fb051c" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 34955 } } peers { permanent_uuid: "30e554990e1d47fb8e2b183647f95d5d" member_type: VOTER last_known_addr { host: "127.13.100.61" port: 46389 } }
I20260929 18:37:49.315701 20369 raft_consensus.cc:374] T 00000000000000000000000000000000 P 30e554990e1d47fb8e2b183647f95d5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:37:49.315733 20369 raft_consensus.cc:729] T 00000000000000000000000000000000 P 30e554990e1d47fb8e2b183647f95d5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30e554990e1d47fb8e2b183647f95d5d, State: Initialized, Role: FOLLOWER
I20260929 18:37:49.316080 20369 consensus_queue.cc:260] T 00000000000000000000000000000000 P 30e554990e1d47fb8e2b183647f95d5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16cac5b4dfe74684861c10d815fb051c" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 34955 } } peers { permanent_uuid: "30e554990e1d47fb8e2b183647f95d5d" member_type: VOTER last_known_addr { host: "127.13.100.61" port: 46389 } }
I20260929 18:37:49.316496 20375 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30e554990e1d47fb8e2b183647f95d5d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16cac5b4dfe74684861c10d815fb051c" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 34955 } } peers { permanent_uuid: "30e554990e1d47fb8e2b183647f95d5d" member_type: VOTER last_known_addr { host: "127.13.100.61" port: 46389 } } }
I20260929 18:37:49.316637 20375 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30e554990e1d47fb8e2b183647f95d5d [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:37:49.317186 20369 sys_catalog.cc:564] T 00000000000000000000000000000000 P 30e554990e1d47fb8e2b183647f95d5d [sys.catalog]: configured and running, proceeding with master startup.
W20260929 18:37:49.321965 20386 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 30e554990e1d47fb8e2b183647f95d5d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:37:49.322045 20386 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:37:49.324239 20301 master_runner.cc:428] Detected that this master 30e554990e1d47fb8e2b183647f95d5d is joining an existing cluster
I20260929 18:37:49.324311 20301 master_runner.cc:432] Initiating AddMaster RPC to add 127.13.100.61:46389
I20260929 18:37:49.336431 20373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:49.326990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48288 (local address 127.13.100.61:46389)
0929 18:37:49.327091 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:49.327095 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:49.327440 (+ 345us) server_negotiation.cc:407] Connection header received
0929 18:37:49.327569 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:49.327573 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:49.327609 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:49.327659 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:49.328180 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:49.328893 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:49.330389 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:49.330701 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:49.330776 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:49.332947 (+ 2171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:49.332954 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:49.332957 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:37:49.332986 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:49.334687 (+ 1701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:49.335120 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:49.335123 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:49.335125 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:49.335206 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:49.335480 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:49.335486 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:49.335490 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:49.335671 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:49.335743 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:49.336123 (+ 380us) server_negotiation.cc:299] Negotiation successful
0929 18:37:49.336255 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":28,"server-negotiator.queue_time_us":42}
I20260929 18:37:49.336450 20393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:49.326920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.61:46389 (local address 127.0.0.1:48288)
0929 18:37:49.327369 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:49.327386 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:49.327486 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:49.327781 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:49.327786 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:49.327952 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:37:49.328084 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:49.328091 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:49.329021 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:49.329023 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:49.330267 (+ 1244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:49.330277 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:49.330933 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:49.330936 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:49.331159 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:49.332418 (+ 1259us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:37:49.332440 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:37:49.332810 (+ 370us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:37:49.334789 (+ 1979us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:49.334792 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:49.334794 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:49.335010 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:49.335306 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:49.335310 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:49.335312 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:49.335393 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:49.335799 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:37:49.335802 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:37:49.335949 (+ 147us) client_negotiation.cc:769] Sending connection context
0929 18:37:49.336179 (+ 230us) client_negotiation.cc:240] Negotiation successful
0929 18:37:49.336308 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":381,"mutex_wait_us":340,"spinlock_wait_cycles":1920,"thread_start_us":287,"threads_started":1}
I20260929 18:37:49.336997 19876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:49.327124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51518 (local address 127.13.100.62:34955)
0929 18:37:49.327218 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:49.327222 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:49.327235 (+ 13us) server_negotiation.cc:407] Connection header received
0929 18:37:49.327275 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:49.327278 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:49.327321 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:49.327404 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:49.328075 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:49.328901 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:49.330395 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:49.330726 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:49.330790 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:49.332657 (+ 1867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:49.332661 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:49.332664 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:37:49.332689 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:49.334433 (+ 1744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:49.334930 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:49.334933 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:49.334935 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:49.335015 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:49.335445 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:49.335451 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:49.335454 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:49.335671 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:49.335760 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:49.336717 (+ 957us) server_negotiation.cc:299] Negotiation successful
0929 18:37:49.336827 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260929 18:37:49.337579 20392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:49.326519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:34955 (local address 127.0.0.1:51518)
0929 18:37:49.326801 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:49.326816 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:49.326934 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:49.327528 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:49.327532 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:49.327808 (+ 276us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:37:49.327942 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:49.327948 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:49.329020 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:49.329023 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:49.330267 (+ 1244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:49.330277 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:49.330933 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:49.330936 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:49.331154 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:49.331940 (+ 786us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:37:49.331957 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:37:49.332460 (+ 503us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:37:49.334559 (+ 2099us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:49.334575 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:49.334577 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:49.334827 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:49.335112 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:49.335115 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:49.335117 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:49.335348 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:49.335799 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:37:49.335802 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:37:49.335962 (+ 160us) client_negotiation.cc:769] Sending connection context
0929 18:37:49.336053 (+ 91us) client_negotiation.cc:240] Negotiation successful
0929 18:37:49.337186 (+ 1133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"mutex_wait_us":287,"spinlock_wait_cycles":46720,"thread_start_us":99,"threads_started":1}
Sep 29 18:37:49 dist-test-slave-j1xs krb5kdc[19792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707068, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.100.62@KRBTEST.COM for oryx/127.13.100.61@KRBTEST.COM
I20260929 18:37:49.351085 20373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:49.341496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48304 (local address 127.13.100.61:46389)
0929 18:37:49.341622 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:49.341626 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:49.341945 (+ 319us) server_negotiation.cc:407] Connection header received
0929 18:37:49.342144 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:49.342146 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:49.342186 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:49.342248 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:49.342941 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:49.343472 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:49.344666 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:49.344884 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:49.344996 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:49.347354 (+ 2358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:49.347357 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:49.347359 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:37:49.347379 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:49.349126 (+ 1747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:49.349597 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:49.349602 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:49.349604 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:49.349682 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:49.350048 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:49.350051 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:49.350053 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:49.350227 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:49.350337 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:49.350741 (+ 404us) server_negotiation.cc:299] Negotiation successful
0929 18:37:49.350871 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 18:37:49.351686 20394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:49.341457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.61:46389 (local address 127.0.0.1:48304)
0929 18:37:49.341812 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:49.341863 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:49.342018 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:49.342305 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:49.342311 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:49.342714 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:37:49.342837 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:49.342844 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:49.343583 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:49.343587 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:49.344546 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:49.344556 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:49.345005 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:49.345008 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:49.345117 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:49.345621 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:37:49.345636 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:37:49.347209 (+ 1573us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:37:49.349243 (+ 2034us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:49.349254 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:49.349261 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:49.349496 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:49.349782 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:49.349785 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:49.349787 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:49.349919 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:49.350337 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:37:49.350355 (+ 18us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:37:49.350484 (+ 129us) client_negotiation.cc:769] Sending connection context
0929 18:37:49.351235 (+ 751us) client_negotiation.cc:240] Negotiation successful
0929 18:37:49.351466 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":110,"threads_started":1}
I20260929 18:37:49.353132 19837 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.13.100.61:46389: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "30e554990e1d47fb8e2b183647f95d5d"
member_type: NON_VOTER
last_known_addr {
host: "127.13.100.61"
port: 46389
}
attrs {
promote: true
}
}
dest_uuid: "16cac5b4dfe74684861c10d815fb051c"
cas_config_opid_index: -1
I20260929 18:37:49.353782 19837 consensus_queue.cc:237] T 00000000000000000000000000000000 P 16cac5b4dfe74684861c10d815fb051c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "16cac5b4dfe74684861c10d815fb051c" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 34955 } } peers { permanent_uuid: "30e554990e1d47fb8e2b183647f95d5d" member_type: NON_VOTER last_known_addr { host: "127.13.100.61" port: 46389 } attrs { promote: true } }
I20260929 18:37:49.355516 20396 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 16cac5b4dfe74684861c10d815fb051c [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 30e554990e1d47fb8e2b183647f95d5d (127.13.100.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "16cac5b4dfe74684861c10d815fb051c" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 34955 } } peers { permanent_uuid: "30e554990e1d47fb8e2b183647f95d5d" member_type: NON_VOTER last_known_addr { host: "127.13.100.61" port: 46389 } attrs { promote: true } } }
I20260929 18:37:49.356401 20343 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 30e554990e1d47fb8e2b183647f95d5d [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:37:49.357746 20396 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.13.100.61:46389
I20260929 18:37:49.357785 20343 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 30e554990e1d47fb8e2b183647f95d5d [term 1 FOLLOWER]: Refusing update from remote peer 16cac5b4dfe74684861c10d815fb051c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
WARNING: no policy specified for oryx/127.13.100.60@KRBTEST.COM; defaulting to no policy
I20260929 18:37:49.358363 20301 master.cc:537] Master@127.13.100.61:46389 shutting down...
I20260929 18:37:49.358500 20397 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 16cac5b4dfe74684861c10d815fb051c [LEADER]: Connected to new peer: Peer: permanent_uuid: "30e554990e1d47fb8e2b183647f95d5d" member_type: NON_VOTER last_known_addr { host: "127.13.100.61" port: 46389 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:37:49.359179 20398 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16cac5b4dfe74684861c10d815fb051c [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "16cac5b4dfe74684861c10d815fb051c" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "16cac5b4dfe74684861c10d815fb051c" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 34955 } } peers { permanent_uuid: "30e554990e1d47fb8e2b183647f95d5d" member_type: NON_VOTER last_known_addr { host: "127.13.100.61" port: 46389 } attrs { promote: true } } }
I20260929 18:37:49.359282 20398 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16cac5b4dfe74684861c10d815fb051c [sys.catalog]: This master's current role is: LEADER
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.100.60@KRBTEST.COM" created.
W20260929 18:37:49.360780 19821 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260929 18:37:49.362828 19821 consensus_peers.cc:487] T 00000000000000000000000000000000 P 16cac5b4dfe74684861c10d815fb051c -> Peer 30e554990e1d47fb8e2b183647f95d5d (127.13.100.61:46389): Couldn't send request to peer 30e554990e1d47fb8e2b183647f95d5d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260929 18:37:49.363181 20301 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 30e554990e1d47fb8e2b183647f95d5d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:37:49.363481 20301 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 30e554990e1d47fb8e2b183647f95d5d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:37:49.363605 20301 tablet_replica.cc:330] T 00000000000000000000000000000000 P 30e554990e1d47fb8e2b183647f95d5d: stopping tablet replica
I20260929 18:37:49.368816 20301 master.cc:552] Master@127.13.100.61:46389 shutdown complete.
I20260929 18:37:49.368919 20301 master_runner.cc:305] Clearing existing system tablet
I20260929 18:37:49.369274 20301 fs_manager.cc:359] Metadata directory not provided
I20260929 18:37:49.369387 20301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:37:49.370909 20301 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:37:49.371942 20403 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:37:49.372231 20301 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:37:49.372380 20301 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:37:49.372591 20301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/master-1/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/master-1/wal
uuid: "30e554990e1d47fb8e2b183647f95d5d"
format_stamp: "Formatted at 2026-09-29 18:37:49 on dist-test-slave-j1xs"
server_key: "1e18e38e77941864ee75094114de4ee6"
server_key_iv: "22b782de56a8e10975aba54c04842d36"
server_key_version: "encryptionkey@0"
I20260929 18:37:49.375089 20301 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 30e554990e1d47fb8e2b183647f95d5d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:37:49.377761 20301 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 30e554990e1d47fb8e2b183647f95d5d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260929 18:37:49.377874 20301 log.cc:1196] T 00000000000000000000000000000000 P 30e554990e1d47fb8e2b183647f95d5d: Deleting WAL directory at /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260929 18:37:49.378170 20301 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 30e554990e1d47fb8e2b183647f95d5d: Deleting consensus metadata
I20260929 18:37:49.378352 20301 master_runner.cc:315] Copying system tablet from 127.13.100.62:34955
I20260929 18:37:49.379539 20301 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 30e554990e1d47fb8e2b183647f95d5d: tablet copy: Beginning tablet copy session from remote peer at address 127.13.100.62:34955
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.100.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/krb5kdc/oryx_127.13.100.60.keytab.
Entry for principal oryx/127.13.100.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/krb5kdc/oryx_127.13.100.60.keytab.
I20260929 18:37:49.380640 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.100.60:40697
--webserver_interface=127.13.100.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/krb5kdc/oryx_127.13.100.60.keytab
--principal=oryx/127.13.100.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.100.20:45575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.100.62:34955,127.13.100.61:46389,127.13.100.60:40697
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 18:37:49.389921 19876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:49.380224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51520 (local address 127.13.100.62:34955)
0929 18:37:49.380361 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:49.380367 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:49.380615 (+ 248us) server_negotiation.cc:407] Connection header received
0929 18:37:49.380831 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:49.380837 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:49.380895 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:49.380963 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:49.381663 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:49.382394 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:49.383625 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:49.384053 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:49.384145 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:49.385764 (+ 1619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:49.385770 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:49.385776 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:37:49.385811 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:49.387683 (+ 1872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:49.388250 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:49.388256 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:49.388260 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:49.388351 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:49.388696 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:49.388702 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:49.388706 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:49.388979 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:49.389067 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:49.389507 (+ 440us) server_negotiation.cc:299] Negotiation successful
0929 18:37:49.389651 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 18:37:49.390642 20409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:49.380115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:34955 (local address 127.0.0.1:51520)
0929 18:37:49.380519 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:49.380534 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:49.380700 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:49.381133 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:49.381138 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:49.381365 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:37:49.381524 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:49.381534 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:49.382515 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:49.382517 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:49.383488 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:49.383501 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:49.384327 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:49.384333 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:49.384455 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:49.385052 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:37:49.385074 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:37:49.385587 (+ 513us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:37:49.387820 (+ 2233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:49.387825 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:49.387830 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:49.388126 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:49.388476 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:49.388481 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:49.388486 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:49.388582 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:49.389147 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:37:49.389153 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:37:49.389291 (+ 138us) client_negotiation.cc:769] Sending connection context
0929 18:37:49.389962 (+ 671us) client_negotiation.cc:240] Negotiation successful
0929 18:37:49.390447 (+ 485us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":87,"threads_started":1}
I20260929 18:37:49.391201 19857 tablet_copy_service.cc:140] P 16cac5b4dfe74684861c10d815fb051c: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 30e554990e1d47fb8e2b183647f95d5d ({username='oryx', principal='oryx/127.13.100.61@KRBTEST.COM'} at 127.0.0.1:51520)
I20260929 18:37:49.391404 19857 tablet_copy_service.cc:161] P 16cac5b4dfe74684861c10d815fb051c: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 30e554990e1d47fb8e2b183647f95d5d at {username='oryx', principal='oryx/127.13.100.61@KRBTEST.COM'} at 127.0.0.1:51520: session id = 30e554990e1d47fb8e2b183647f95d5d-00000000000000000000000000000000
I20260929 18:37:49.393558 19857 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 16cac5b4dfe74684861c10d815fb051c: Tablet Copy: opened 0 blocks and 1 log segments
I20260929 18:37:49.395308 20301 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260929 18:37:49.397697 20301 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 30e554990e1d47fb8e2b183647f95d5d: tablet copy: Starting download of 0 data blocks...
I20260929 18:37:49.397939 20301 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 30e554990e1d47fb8e2b183647f95d5d: tablet copy: Starting download of 1 WAL segments...
I20260929 18:37:49.399734 20301 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 30e554990e1d47fb8e2b183647f95d5d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260929 18:37:49.401633 19857 tablet_copy_service.cc:342] P 16cac5b4dfe74684861c10d815fb051c: Request end of tablet copy session 30e554990e1d47fb8e2b183647f95d5d-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.13.100.61@KRBTEST.COM'} at 127.0.0.1:51520
I20260929 18:37:49.401809 19857 tablet_copy_service.cc:433] P 16cac5b4dfe74684861c10d815fb051c: ending tablet copy session 30e554990e1d47fb8e2b183647f95d5d-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 30e554990e1d47fb8e2b183647f95d5d
I20260929 18:37:49.409618 20301 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:37:49.411648 20414 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:37:49.411649 20411 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:37:49.411955 20301 server_base.cc:1034] running on GCE node
W20260929 18:37:49.412133 20412 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:37:49.412417 20301 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:37:49.412707 20301 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:37:49.413859 20301 hybrid_clock.cc:648] HybridClock initialized: now 1790707069413834 us; error 42 us; skew 500 ppm
Sep 29 18:37:49 dist-test-slave-j1xs krb5kdc[19792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707069, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.100.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:37:49.415441 20301 init.cc:377] Logged in from keytab as oryx/127.13.100.61@KRBTEST.COM (short username oryx)
I20260929 18:37:49.415867 20301 webserver.cc:460] Webserver started at http://127.13.100.61:37727/ using document root <none> and password file <none>
I20260929 18:37:49.416061 20301 fs_manager.cc:359] Metadata directory not provided
I20260929 18:37:49.416103 20301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:37:49.417405 20301 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:37:49.418469 20421 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:37:49.418746 20301 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260929 18:37:49.418819 20301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/master-1/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/master-1/wal
uuid: "30e554990e1d47fb8e2b183647f95d5d"
format_stamp: "Formatted at 2026-09-29 18:37:49 on dist-test-slave-j1xs"
server_key: "1e18e38e77941864ee75094114de4ee6"
server_key_iv: "22b782de56a8e10975aba54c04842d36"
server_key_version: "encryptionkey@0"
I20260929 18:37:49.418896 20301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:37:49.438388 20301 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:37:49.440496 20301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:37:49.445263 20301 rpc_server.cc:307] RPC server started. Bound to: 127.13.100.61:46389
I20260929 18:37:49.445542 20473 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.100.61:46389 every 8 connection(s)
I20260929 18:37:49.445874 20301 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/master-1/data/info.pb
I20260929 18:37:49.447654 20474 sys_catalog.cc:263] Verifying existing consensus state
I20260929 18:37:49.448657 20474 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30e554990e1d47fb8e2b183647f95d5d: Bootstrap starting.
I20260929 18:37:49.458611 20474 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30e554990e1d47fb8e2b183647f95d5d: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 18:37:49.458933 20474 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30e554990e1d47fb8e2b183647f95d5d: Bootstrap complete.
I20260929 18:37:49.459503 20474 raft_consensus.cc:348] T 00000000000000000000000000000000 P 30e554990e1d47fb8e2b183647f95d5d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "16cac5b4dfe74684861c10d815fb051c" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 34955 } } peers { permanent_uuid: "30e554990e1d47fb8e2b183647f95d5d" member_type: NON_VOTER last_known_addr { host: "127.13.100.61" port: 46389 } attrs { promote: true } }
I20260929 18:37:49.459667 20474 raft_consensus.cc:729] T 00000000000000000000000000000000 P 30e554990e1d47fb8e2b183647f95d5d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 30e554990e1d47fb8e2b183647f95d5d, State: Initialized, Role: LEARNER
I20260929 18:37:49.459784 20474 consensus_queue.cc:260] T 00000000000000000000000000000000 P 30e554990e1d47fb8e2b183647f95d5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "16cac5b4dfe74684861c10d815fb051c" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 34955 } } peers { permanent_uuid: "30e554990e1d47fb8e2b183647f95d5d" member_type: NON_VOTER last_known_addr { host: "127.13.100.61" port: 46389 } attrs { promote: true } }
I20260929 18:37:49.460148 20477 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30e554990e1d47fb8e2b183647f95d5d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "16cac5b4dfe74684861c10d815fb051c" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 34955 } } peers { permanent_uuid: "30e554990e1d47fb8e2b183647f95d5d" member_type: NON_VOTER last_known_addr { host: "127.13.100.61" port: 46389 } attrs { promote: true } } }
I20260929 18:37:49.460256 20474 sys_catalog.cc:564] T 00000000000000000000000000000000 P 30e554990e1d47fb8e2b183647f95d5d [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:37:49.460345 20477 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30e554990e1d47fb8e2b183647f95d5d [sys.catalog]: This master's current role is: LEARNER
I20260929 18:37:49.463562 20488 catalog_manager.cc:1259] Loaded cluster ID: 3dd0e53705814f59aa70862dfbe1aea6
I20260929 18:37:49.463688 20488 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 30e554990e1d47fb8e2b183647f95d5d: loading cluster ID for follower catalog manager: success
I20260929 18:37:49.465849 20488 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 30e554990e1d47fb8e2b183647f95d5d: acquiring CA information for follower catalog manager: success
I20260929 18:37:49.467408 20488 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 30e554990e1d47fb8e2b183647f95d5d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260929 18:37:49.510831 20410 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:37:49.511130 20410 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:37:49.511205 20410 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:37:49.515244 20410 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:37:49.515323 20410 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:37:49.515360 20410 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:37:49.515396 20410 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:37:49.515430 20410 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:37:49.520992 20410 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.100.20:45575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.13.100.62:34955,127.13.100.61:46389,127.13.100.60:40697
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/krb5kdc/oryx_127.13.100.60.keytab
--principal=oryx/127.13.100.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.100.60:40697
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.13.100.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20410
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:37:49.522353 20410 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:37:49.523404 20410 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:37:49.531690 20496 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:37:49.531687 20493 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:37:49.531690 20494 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:37:49.531738 20410 server_base.cc:1034] running on GCE node
I20260929 18:37:49.532471 20410 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:37:49.533610 20410 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:37:49.534802 20410 hybrid_clock.cc:648] HybridClock initialized: now 1790707069534757 us; error 58 us; skew 500 ppm
Sep 29 18:37:49 dist-test-slave-j1xs krb5kdc[19792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707069, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.100.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:37:49.542807 20410 init.cc:377] Logged in from keytab as oryx/127.13.100.60@KRBTEST.COM (short username oryx)
I20260929 18:37:49.543715 20410 webserver.cc:460] Webserver started at http://127.13.100.60:39701/ using document root <none> and password file <none>
I20260929 18:37:49.544224 20410 fs_manager.cc:359] Metadata directory not provided
I20260929 18:37:49.544282 20410 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:37:49.544440 20410 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:37:49.546522 20410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/master-2/data/instance:
uuid: "cdfc2d79001d4a34ae6eddb06d8a0c0d"
format_stamp: "Formatted at 2026-09-29 18:37:49 on dist-test-slave-j1xs"
server_key: "38298185906dff45f48c4954a86215c3"
server_key_iv: "a84c6fe361ac8c1e10f5e9783ed47e4f"
server_key_version: "encryptionkey@0"
I20260929 18:37:49.547479 20410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/master-2/wal/instance:
uuid: "cdfc2d79001d4a34ae6eddb06d8a0c0d"
format_stamp: "Formatted at 2026-09-29 18:37:49 on dist-test-slave-j1xs"
server_key: "38298185906dff45f48c4954a86215c3"
server_key_iv: "a84c6fe361ac8c1e10f5e9783ed47e4f"
server_key_version: "encryptionkey@0"
I20260929 18:37:49.551120 20410 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 18:37:49.553483 20503 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:37:49.554747 20410 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:37:49.554867 20410 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/master-2/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/master-2/wal
uuid: "cdfc2d79001d4a34ae6eddb06d8a0c0d"
format_stamp: "Formatted at 2026-09-29 18:37:49 on dist-test-slave-j1xs"
server_key: "38298185906dff45f48c4954a86215c3"
server_key_iv: "a84c6fe361ac8c1e10f5e9783ed47e4f"
server_key_version: "encryptionkey@0"
I20260929 18:37:49.554970 20410 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:37:49.568549 20410 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:37:49.571673 20410 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:37:49.571857 20410 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:37:49.578413 20410 rpc_server.cc:307] RPC server started. Bound to: 127.13.100.60:40697
I20260929 18:37:49.578758 20555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.100.60:40697 every 8 connection(s)
I20260929 18:37:49.579363 20410 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/master-2/data/info.pb
I20260929 18:37:49.582079 20556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:37:49.585474 20556 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.100.62" port: 34955 } has no permanent_uuid. Determining permanent_uuid...
I20260929 18:37:49.588145 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 20410
Sep 29 18:37:49 dist-test-slave-j1xs krb5kdc[19792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707069, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.100.60@KRBTEST.COM for oryx/127.13.100.62@KRBTEST.COM
Sep 29 18:37:49 dist-test-slave-j1xs krb5kdc[19792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707068, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.13.100.60@KRBTEST.COM
I20260929 18:37:49.599916 19876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:49.587466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51522 (local address 127.13.100.62:34955)
0929 18:37:49.587608 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:49.587612 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:49.588380 (+ 768us) server_negotiation.cc:407] Connection header received
0929 18:37:49.589443 (+ 1063us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:49.589447 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:49.589502 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:49.589598 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:49.590831 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:49.591681 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:49.592960 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:49.593238 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:49.593341 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:49.596113 (+ 2772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:49.596117 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:49.596125 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:37:49.596147 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:49.597739 (+ 1592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:49.598381 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:49.598384 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:49.598385 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:49.598454 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:49.598799 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:49.598807 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:49.598809 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:49.598968 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:49.599066 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:49.599637 (+ 571us) server_negotiation.cc:299] Negotiation successful
0929 18:37:49.599748 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 18:37:49.600709 20558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:49.587695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:34955 (local address 127.0.0.1:51522)
0929 18:37:49.588220 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:49.588277 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:49.589250 (+ 973us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:49.589772 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:49.589779 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:49.590233 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:37:49.590673 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:49.590693 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:49.591809 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:49.591812 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:49.592846 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:49.592856 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:49.593371 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:49.593374 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:49.593550 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:49.594202 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:37:49.594225 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:37:49.595941 (+ 1716us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:37:49.597843 (+ 1902us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:49.597850 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:49.597860 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:49.598279 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:49.598550 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:49.598555 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:49.598557 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:49.598698 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:49.599075 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:37:49.599081 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:37:49.599395 (+ 314us) client_negotiation.cc:769] Sending connection context
0929 18:37:49.599585 (+ 190us) client_negotiation.cc:240] Negotiation successful
0929 18:37:49.599865 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"mutex_wait_us":47,"thread_start_us":145,"threads_started":1}
I20260929 18:37:49.600713 20559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:49.588924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54418 (local address 127.13.100.60:40697)
0929 18:37:49.589662 (+ 738us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:49.589677 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:49.589705 (+ 28us) server_negotiation.cc:407] Connection header received
0929 18:37:49.589750 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:49.589753 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:49.589965 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:49.590106 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:49.590946 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:49.592073 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:49.593126 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:49.593605 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:49.593681 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:49.596516 (+ 2835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:49.596521 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:49.596531 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:37:49.596554 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:49.598352 (+ 1798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:49.598750 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:49.598755 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:49.598757 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:49.598833 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:49.599112 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:49.599115 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:49.599117 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:49.599367 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:49.599562 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:49.599783 (+ 221us) server_negotiation.cc:299] Negotiation successful
0929 18:37:49.599921 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":654,"thread_start_us":541,"threads_started":1}
I20260929 18:37:49.602569 20556 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.100.61" port: 46389 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 18:37:49 dist-test-slave-j1xs krb5kdc[19792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707069, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.100.60@KRBTEST.COM for oryx/127.13.100.61@KRBTEST.COM
I20260929 18:37:49.610672 20558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:49.602986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.61:46389 (local address 127.0.0.1:48306)
0929 18:37:49.603310 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:49.603321 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:49.603378 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:49.603596 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:49.603598 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:49.603713 (+ 115us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:37:49.603791 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:49.603795 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:49.604642 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:49.604644 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:49.605203 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:49.605211 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:49.605625 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:49.605628 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:49.605712 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:49.606260 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:37:49.606273 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:37:49.607459 (+ 1186us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:37:49.609156 (+ 1697us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:49.609159 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:49.609161 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:49.609361 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:49.609689 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:49.609692 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:49.609693 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:49.609797 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:49.610230 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:37:49.610232 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:37:49.610314 (+ 82us) client_negotiation.cc:769] Sending connection context
0929 18:37:49.610410 (+ 96us) client_negotiation.cc:240] Negotiation successful
0929 18:37:49.610556 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272}
I20260929 18:37:49.610697 20560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:49.603109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48306 (local address 127.13.100.61:46389)
0929 18:37:49.603344 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:49.603347 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:49.603360 (+ 13us) server_negotiation.cc:407] Connection header received
0929 18:37:49.603444 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:49.603446 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:49.603482 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:49.603582 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:49.603879 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:49.604549 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:49.605298 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:49.605511 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:49.605603 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:49.607604 (+ 2001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:49.607607 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:49.607609 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:37:49.607629 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:49.609035 (+ 1406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:49.609484 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:49.609487 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:49.609489 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:49.609562 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:49.609947 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:49.609950 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:49.609952 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:49.610138 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:49.610216 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:49.610460 (+ 244us) server_negotiation.cc:299] Negotiation successful
0929 18:37:49.610555 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":97,"threads_started":1}
I20260929 18:37:49.611498 20556 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.100.60" port: 40697 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 18:37:49 dist-test-slave-j1xs krb5kdc[19792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707069, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.100.60@KRBTEST.COM for oryx/127.13.100.60@KRBTEST.COM
I20260929 18:37:49.619140 20558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:49.611978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.60:40697 (local address 127.0.0.1:54420)
0929 18:37:49.612086 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:49.612102 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:49.612171 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:49.612387 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:49.612390 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:49.612540 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:37:49.612603 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:49.612607 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:49.613314 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:49.613317 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:49.613808 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:49.613815 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:49.614306 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:49.614309 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:49.614380 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:49.614836 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:37:49.614847 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:37:49.615935 (+ 1088us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:37:49.617764 (+ 1829us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:49.617767 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:49.617768 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:49.618016 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:49.618284 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:49.618286 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:49.618287 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:49.618355 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:49.618757 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:37:49.618759 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:37:49.618828 (+ 69us) client_negotiation.cc:769] Sending connection context
0929 18:37:49.618909 (+ 81us) client_negotiation.cc:240] Negotiation successful
0929 18:37:49.619015 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260929 18:37:49.619318 20559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:49.612094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54420 (local address 127.13.100.60:40697)
0929 18:37:49.612172 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:49.612176 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:49.612188 (+ 12us) server_negotiation.cc:407] Connection header received
0929 18:37:49.612257 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:49.612260 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:49.612291 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:49.612380 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:49.612705 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:49.613198 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:49.613940 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:49.614203 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:49.614311 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:49.616058 (+ 1747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:49.616061 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:49.616063 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:37:49.616088 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:49.617655 (+ 1567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:49.618118 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:49.618121 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:49.618122 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:49.618194 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:49.618483 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:49.618486 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:49.618487 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:49.618674 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:49.618753 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:49.618967 (+ 214us) server_negotiation.cc:299] Negotiation successful
0929 18:37:49.619115 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260929 18:37:49.622293 20556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cdfc2d79001d4a34ae6eddb06d8a0c0d: Bootstrap starting.
I20260929 18:37:49.624749 20556 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cdfc2d79001d4a34ae6eddb06d8a0c0d: Neither blocks nor log segments found. Creating new log.
I20260929 18:37:49.625447 20556 log.cc:824] T 00000000000000000000000000000000 P cdfc2d79001d4a34ae6eddb06d8a0c0d: Log is configured to *not* fsync() on all Append() calls
I20260929 18:37:49.627396 20556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cdfc2d79001d4a34ae6eddb06d8a0c0d: No bootstrap required, opened a new log
I20260929 18:37:49.630333 20556 raft_consensus.cc:348] T 00000000000000000000000000000000 P cdfc2d79001d4a34ae6eddb06d8a0c0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16cac5b4dfe74684861c10d815fb051c" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 34955 } } peers { permanent_uuid: "30e554990e1d47fb8e2b183647f95d5d" member_type: VOTER last_known_addr { host: "127.13.100.61" port: 46389 } } peers { permanent_uuid: "cdfc2d79001d4a34ae6eddb06d8a0c0d" member_type: VOTER last_known_addr { host: "127.13.100.60" port: 40697 } }
I20260929 18:37:49.630561 20556 raft_consensus.cc:374] T 00000000000000000000000000000000 P cdfc2d79001d4a34ae6eddb06d8a0c0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:37:49.630642 20556 raft_consensus.cc:729] T 00000000000000000000000000000000 P cdfc2d79001d4a34ae6eddb06d8a0c0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdfc2d79001d4a34ae6eddb06d8a0c0d, State: Initialized, Role: FOLLOWER
I20260929 18:37:49.631096 20556 consensus_queue.cc:260] T 00000000000000000000000000000000 P cdfc2d79001d4a34ae6eddb06d8a0c0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16cac5b4dfe74684861c10d815fb051c" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 34955 } } peers { permanent_uuid: "30e554990e1d47fb8e2b183647f95d5d" member_type: VOTER last_known_addr { host: "127.13.100.61" port: 46389 } } peers { permanent_uuid: "cdfc2d79001d4a34ae6eddb06d8a0c0d" member_type: VOTER last_known_addr { host: "127.13.100.60" port: 40697 } }
I20260929 18:37:49.631569 20562 sys_catalog.cc:455] T 00000000000000000000000000000000 P cdfc2d79001d4a34ae6eddb06d8a0c0d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16cac5b4dfe74684861c10d815fb051c" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 34955 } } peers { permanent_uuid: "30e554990e1d47fb8e2b183647f95d5d" member_type: VOTER last_known_addr { host: "127.13.100.61" port: 46389 } } peers { permanent_uuid: "cdfc2d79001d4a34ae6eddb06d8a0c0d" member_type: VOTER last_known_addr { host: "127.13.100.60" port: 40697 } } }
I20260929 18:37:49.631691 20562 sys_catalog.cc:458] T 00000000000000000000000000000000 P cdfc2d79001d4a34ae6eddb06d8a0c0d [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:37:49.632252 20556 sys_catalog.cc:564] T 00000000000000000000000000000000 P cdfc2d79001d4a34ae6eddb06d8a0c0d [sys.catalog]: configured and running, proceeding with master startup.
W20260929 18:37:49.636955 20573 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cdfc2d79001d4a34ae6eddb06d8a0c0d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:37:49.637048 20573 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:37:49.637724 20028 heartbeater.cc:503] Master 127.13.100.62:34955 was elected leader, sending a full tablet report...
I20260929 18:37:49.640522 20410 master_runner.cc:428] Detected that this master cdfc2d79001d4a34ae6eddb06d8a0c0d is joining an existing cluster
I20260929 18:37:49.640600 20410 master_runner.cc:432] Initiating AddMaster RPC to add 127.13.100.60:40697
I20260929 18:37:49.652482 19876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:49.643534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51532 (local address 127.13.100.62:34955)
0929 18:37:49.643663 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:49.643667 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:49.643677 (+ 10us) server_negotiation.cc:407] Connection header received
0929 18:37:49.643707 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:49.643710 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:49.643743 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:49.643832 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:49.644395 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:49.645194 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:49.646109 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:49.646336 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:49.646446 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:49.647752 (+ 1306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:49.647756 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:49.647759 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:37:49.647787 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:49.649531 (+ 1744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:49.651211 (+ 1680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:49.651215 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:49.651217 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:49.651300 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:49.651590 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:49.651593 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:49.651594 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:49.651805 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:49.651870 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:49.652197 (+ 327us) server_negotiation.cc:299] Negotiation successful
0929 18:37:49.652308 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 18:37:49.652550 20560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:49.643531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48322 (local address 127.13.100.61:46389)
0929 18:37:49.644013 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:49.644017 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:49.644609 (+ 592us) server_negotiation.cc:407] Connection header received
0929 18:37:49.644678 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:49.644681 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:49.644729 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:49.644814 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:49.645274 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:49.646070 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:49.647309 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:49.647641 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:49.647713 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:49.649090 (+ 1377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:49.649093 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:49.649097 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:37:49.649124 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:49.650796 (+ 1672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:49.651318 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:49.651322 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:49.651324 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:49.651402 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:49.651698 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:49.651701 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:49.651703 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:49.651894 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:49.651958 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:49.652278 (+ 320us) server_negotiation.cc:299] Negotiation successful
0929 18:37:49.652397 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384}
I20260929 18:37:49.652959 20579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:49.643095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:34955 (local address 127.0.0.1:51532)
0929 18:37:49.643359 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:49.643376 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:49.643488 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:49.643846 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:49.643849 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:49.644109 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:37:49.644253 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:49.644264 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:49.645325 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:49.645327 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:49.645994 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:49.646008 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:49.646463 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:49.646466 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:49.646591 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:49.647182 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:37:49.647196 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:37:49.647628 (+ 432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:37:49.649923 (+ 2295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:49.649930 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:49.649935 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:49.651106 (+ 1171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:49.651403 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:49.651405 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:49.651407 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:49.651499 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:49.651935 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:37:49.651937 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:37:49.652019 (+ 82us) client_negotiation.cc:769] Sending connection context
0929 18:37:49.652637 (+ 618us) client_negotiation.cc:240] Negotiation successful
0929 18:37:49.652785 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"mutex_wait_us":1,"thread_start_us":94,"threads_started":1}
I20260929 18:37:49.653092 20581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:49.643162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.61:46389 (local address 127.0.0.1:48322)
0929 18:37:49.644508 (+ 1346us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:49.644521 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:49.644607 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:49.644841 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:49.644844 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:49.645035 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:37:49.645147 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:49.645153 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:49.646212 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:49.646215 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:49.647178 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:49.647188 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:49.647890 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:49.647894 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:49.647983 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:49.648537 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:37:49.648551 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:37:49.648948 (+ 397us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:37:49.650902 (+ 1954us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:49.650905 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:49.650907 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:49.651207 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:49.651508 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:49.651510 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:49.651513 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:49.651600 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:49.652024 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:37:49.652027 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:37:49.652104 (+ 77us) client_negotiation.cc:769] Sending connection context
0929 18:37:49.652727 (+ 623us) client_negotiation.cc:240] Negotiation successful
0929 18:37:49.652857 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1290,"mutex_wait_us":26,"thread_start_us":97,"threads_started":1}
I20260929 18:37:49.654238 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 19899
I20260929 18:37:49.662774 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.100.1:38831
--local_ip_for_outbound_sockets=127.13.100.1
--tserver_master_addrs=127.13.100.62:34955
--webserver_port=46753
--webserver_interface=127.13.100.1
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/krb5kdc/oryx_127.13.100.1.keytab
--principal=oryx/127.13.100.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.100.20:45575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.13.100.62:34955,127.13.100.61:46389
--tserver_master_addrs=127.13.100.62:34955,127.13.100.61:46389,127.13.100.60:40697 with env {KRB5CCNAME=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 18:37:49 dist-test-slave-j1xs krb5kdc[19792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707068, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.100.62@KRBTEST.COM for oryx/127.13.100.60@KRBTEST.COM
I20260929 18:37:49.682530 20394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:49.656867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.60:40697 (local address 127.0.0.1:54444)
0929 18:37:49.656972 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:49.656986 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:49.657057 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:49.662677 (+ 5620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:49.662681 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:49.662935 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:37:49.663064 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:49.663073 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:49.663866 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:49.663873 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:49.664818 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:49.664832 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:49.665523 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:49.665527 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:49.665626 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:49.666235 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:37:49.666250 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:37:49.667747 (+ 1497us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:37:49.680674 (+ 12927us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:49.680679 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:49.680681 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:49.680980 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:49.681350 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:49.681354 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:49.681356 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:49.681451 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:49.681966 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:37:49.681970 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:37:49.682076 (+ 106us) client_negotiation.cc:769] Sending connection context
0929 18:37:49.682167 (+ 91us) client_negotiation.cc:240] Negotiation successful
0929 18:37:49.682331 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260929 18:37:49.682955 20559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:49.644296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54432 (local address 127.13.100.60:40697)
0929 18:37:49.644392 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:49.644396 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:49.650230 (+ 5834us) server_negotiation.cc:407] Connection header received
0929 18:37:49.650334 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:49.650337 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:49.650377 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:49.650455 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:49.650990 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:49.651694 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:49.658295 (+ 6601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:49.658663 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:49.658728 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:49.679352 (+ 20624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:49.679356 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:49.679359 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:37:49.679380 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:49.681424 (+ 2044us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:49.681845 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:49.681848 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:49.681850 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:49.681947 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:49.682205 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:49.682208 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:49.682210 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:49.682390 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:49.682463 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:49.682707 (+ 244us) server_negotiation.cc:299] Negotiation successful
0929 18:37:49.682808 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 18:37:49.683414 20580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:49.644218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.60:40697 (local address 127.0.0.1:54432)
0929 18:37:49.650126 (+ 5908us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:49.650141 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:49.650247 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:49.650490 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:49.650495 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:49.650735 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:37:49.650863 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:49.650873 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:49.651805 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:49.651809 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:49.658176 (+ 6367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:49.658187 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:49.659003 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:49.659006 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:49.659091 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:49.678630 (+ 19539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:37:49.678647 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:37:49.679195 (+ 548us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:37:49.681537 (+ 2342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:49.681540 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:49.681542 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:49.681756 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:49.682038 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:49.682040 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:49.682042 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:49.682111 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:49.682502 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:37:49.682504 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:37:49.682570 (+ 66us) client_negotiation.cc:769] Sending connection context
0929 18:37:49.683132 (+ 562us) client_negotiation.cc:240] Negotiation successful
0929 18:37:49.683263 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5819,"thread_start_us":101,"threads_started":1}
I20260929 18:37:49.684527 20582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:49.662100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54444 (local address 127.13.100.60:40697)
0929 18:37:49.662409 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:49.662418 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:49.662434 (+ 16us) server_negotiation.cc:407] Connection header received
0929 18:37:49.662495 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:49.662497 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:49.662542 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:49.662658 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:49.663220 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:49.663743 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:49.664999 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:49.665391 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:49.665523 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:49.667902 (+ 2379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:49.667908 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:49.667913 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:37:49.667955 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:49.680520 (+ 12565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:49.681130 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:49.681133 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:49.681135 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:49.681233 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:49.681571 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:49.681574 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:49.681576 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:49.681812 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:49.683325 (+ 1513us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:49.683408 (+ 83us) server_negotiation.cc:299] Negotiation successful
0929 18:37:49.684349 (+ 941us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":119,"threads_started":1}
I20260929 18:37:49.685623 19837 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.13.100.60:40697: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "cdfc2d79001d4a34ae6eddb06d8a0c0d"
member_type: NON_VOTER
last_known_addr {
host: "127.13.100.60"
port: 40697
}
attrs {
promote: true
}
}
dest_uuid: "16cac5b4dfe74684861c10d815fb051c"
cas_config_opid_index: 5
I20260929 18:37:49.686280 19837 consensus_queue.cc:237] T 00000000000000000000000000000000 P 16cac5b4dfe74684861c10d815fb051c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "16cac5b4dfe74684861c10d815fb051c" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 34955 } } peers { permanent_uuid: "30e554990e1d47fb8e2b183647f95d5d" member_type: NON_VOTER last_known_addr { host: "127.13.100.61" port: 46389 } attrs { promote: true } } peers { permanent_uuid: "cdfc2d79001d4a34ae6eddb06d8a0c0d" member_type: NON_VOTER last_known_addr { host: "127.13.100.60" port: 40697 } attrs { promote: true } }
I20260929 18:37:49.687448 20398 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 16cac5b4dfe74684861c10d815fb051c [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER cdfc2d79001d4a34ae6eddb06d8a0c0d (127.13.100.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "16cac5b4dfe74684861c10d815fb051c" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 34955 } } peers { permanent_uuid: "30e554990e1d47fb8e2b183647f95d5d" member_type: NON_VOTER last_known_addr { host: "127.13.100.61" port: 46389 } attrs { promote: true } } peers { permanent_uuid: "cdfc2d79001d4a34ae6eddb06d8a0c0d" member_type: NON_VOTER last_known_addr { host: "127.13.100.60" port: 40697 } attrs { promote: true } } }
I20260929 18:37:49.688423 20530 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cdfc2d79001d4a34ae6eddb06d8a0c0d [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:37:49.688871 20398 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.13.100.60:40697
I20260929 18:37:49.689021 20398 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16cac5b4dfe74684861c10d815fb051c [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "16cac5b4dfe74684861c10d815fb051c" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "16cac5b4dfe74684861c10d815fb051c" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 34955 } } peers { permanent_uuid: "30e554990e1d47fb8e2b183647f95d5d" member_type: NON_VOTER last_known_addr { host: "127.13.100.61" port: 46389 } attrs { promote: true } } peers { permanent_uuid: "cdfc2d79001d4a34ae6eddb06d8a0c0d" member_type: NON_VOTER last_known_addr { host: "127.13.100.60" port: 40697 } attrs { promote: true } } }
I20260929 18:37:49.689106 20398 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16cac5b4dfe74684861c10d815fb051c [sys.catalog]: This master's current role is: LEADER
I20260929 18:37:49.689507 20410 master.cc:537] Master@127.13.100.60:40697 shutting down...
I20260929 18:37:49.689952 20530 raft_consensus.cc:1261] T 00000000000000000000000000000000 P cdfc2d79001d4a34ae6eddb06d8a0c0d [term 1 FOLLOWER]: Refusing update from remote peer 16cac5b4dfe74684861c10d815fb051c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260929 18:37:49.691184 20397 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 16cac5b4dfe74684861c10d815fb051c [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdfc2d79001d4a34ae6eddb06d8a0c0d" member_type: NON_VOTER last_known_addr { host: "127.13.100.60" port: 40697 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 18:37:49.693470 19824 consensus_peers.cc:487] T 00000000000000000000000000000000 P 16cac5b4dfe74684861c10d815fb051c -> Peer cdfc2d79001d4a34ae6eddb06d8a0c0d (127.13.100.60:40697): Couldn't send request to peer cdfc2d79001d4a34ae6eddb06d8a0c0d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260929 18:37:49.696031 20560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:49.687512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48330 (local address 127.13.100.61:46389)
0929 18:37:49.687633 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:49.687638 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:49.690806 (+ 3168us) server_negotiation.cc:407] Connection header received
0929 18:37:49.690871 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:49.690874 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:49.690962 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:49.691056 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:37:49.691559 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:49.692349 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:49.694192 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:49.694870 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:49.695005 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:49.695154 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:49.695783 (+ 629us) server_negotiation.cc:299] Negotiation successful
0929 18:37:49.695888 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 18:37:49.696203 20410 raft_consensus.cc:2229] T 00000000000000000000000000000000 P cdfc2d79001d4a34ae6eddb06d8a0c0d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:37:49.696446 20410 raft_consensus.cc:2258] T 00000000000000000000000000000000 P cdfc2d79001d4a34ae6eddb06d8a0c0d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:37:49.696453 20394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:49.687314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.61:46389 (local address 127.0.0.1:48330)
0929 18:37:49.690598 (+ 3284us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:49.690615 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:49.690706 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:49.691199 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:49.691202 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:49.691220 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:37:49.691316 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:49.691323 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:49.692499 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:49.692503 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:49.694037 (+ 1534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:49.694049 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:49.695221 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:49.695226 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:49.695321 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:49.695367 (+ 46us) client_negotiation.cc:769] Sending connection context
0929 18:37:49.695450 (+ 83us) client_negotiation.cc:240] Negotiation successful
0929 18:37:49.695525 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3203}
I20260929 18:37:49.696544 20410 tablet_replica.cc:330] T 00000000000000000000000000000000 P cdfc2d79001d4a34ae6eddb06d8a0c0d: stopping tablet replica
I20260929 18:37:49.697278 20448 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 30e554990e1d47fb8e2b183647f95d5d [term 1 LEARNER]: Refusing update from remote peer 16cac5b4dfe74684861c10d815fb051c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260929 18:37:49.697813 20397 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 16cac5b4dfe74684861c10d815fb051c [LEADER]: Connected to new peer: Peer: permanent_uuid: "30e554990e1d47fb8e2b183647f95d5d" member_type: NON_VOTER last_known_addr { host: "127.13.100.61" port: 46389 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260929 18:37:49.699683 20448 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 30e554990e1d47fb8e2b183647f95d5d [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER cdfc2d79001d4a34ae6eddb06d8a0c0d (127.13.100.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "16cac5b4dfe74684861c10d815fb051c" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 34955 } } peers { permanent_uuid: "30e554990e1d47fb8e2b183647f95d5d" member_type: NON_VOTER last_known_addr { host: "127.13.100.61" port: 46389 } attrs { promote: true } } peers { permanent_uuid: "cdfc2d79001d4a34ae6eddb06d8a0c0d" member_type: NON_VOTER last_known_addr { host: "127.13.100.60" port: 40697 } attrs { promote: true } } }
I20260929 18:37:49.702181 20477 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30e554990e1d47fb8e2b183647f95d5d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 16cac5b4dfe74684861c10d815fb051c. Latest consensus state: current_term: 1 leader_uuid: "16cac5b4dfe74684861c10d815fb051c" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "16cac5b4dfe74684861c10d815fb051c" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 34955 } } peers { permanent_uuid: "30e554990e1d47fb8e2b183647f95d5d" member_type: NON_VOTER last_known_addr { host: "127.13.100.61" port: 46389 } attrs { promote: true } } peers { permanent_uuid: "cdfc2d79001d4a34ae6eddb06d8a0c0d" member_type: NON_VOTER last_known_addr { host: "127.13.100.60" port: 40697 } attrs { promote: true } } }
I20260929 18:37:49.702323 20477 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30e554990e1d47fb8e2b183647f95d5d [sys.catalog]: This master's current role is: LEARNER
I20260929 18:37:49.702432 20477 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30e554990e1d47fb8e2b183647f95d5d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "16cac5b4dfe74684861c10d815fb051c" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "16cac5b4dfe74684861c10d815fb051c" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 34955 } } peers { permanent_uuid: "30e554990e1d47fb8e2b183647f95d5d" member_type: NON_VOTER last_known_addr { host: "127.13.100.61" port: 46389 } attrs { promote: true } } peers { permanent_uuid: "cdfc2d79001d4a34ae6eddb06d8a0c0d" member_type: NON_VOTER last_known_addr { host: "127.13.100.60" port: 40697 } attrs { promote: true } } }
I20260929 18:37:49.702513 20477 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30e554990e1d47fb8e2b183647f95d5d [sys.catalog]: This master's current role is: LEARNER
I20260929 18:37:49.703406 20397 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 16cac5b4dfe74684861c10d815fb051c: attempting to promote NON_VOTER 30e554990e1d47fb8e2b183647f95d5d to VOTER
I20260929 18:37:49.703878 20397 consensus_queue.cc:237] T 00000000000000000000000000000000 P 16cac5b4dfe74684861c10d815fb051c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "16cac5b4dfe74684861c10d815fb051c" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 34955 } } peers { permanent_uuid: "30e554990e1d47fb8e2b183647f95d5d" member_type: VOTER last_known_addr { host: "127.13.100.61" port: 46389 } attrs { promote: false } } peers { permanent_uuid: "cdfc2d79001d4a34ae6eddb06d8a0c0d" member_type: NON_VOTER last_known_addr { host: "127.13.100.60" port: 40697 } attrs { promote: true } }
I20260929 18:37:49.705078 20394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:49.704805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.60:40697 (local address 127.0.0.1:54452)
0929 18:37:49.704911 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:49.705000 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.60:40697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260929 18:37:49.705209 20448 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 30e554990e1d47fb8e2b183647f95d5d [term 1 LEARNER]: Refusing update from remote peer 16cac5b4dfe74684861c10d815fb051c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260929 18:37:49.705581 20398 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 16cac5b4dfe74684861c10d815fb051c [LEADER]: Connected to new peer: Peer: permanent_uuid: "30e554990e1d47fb8e2b183647f95d5d" member_type: VOTER last_known_addr { host: "127.13.100.61" port: 46389 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260929 18:37:49.706115 20394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:49.705765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.60:40697 (local address 127.0.0.1:54468)
0929 18:37:49.705999 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:49.706050 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.60:40697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170}
W20260929 18:37:49.706305 19824 consensus_peers.cc:487] T 00000000000000000000000000000000 P 16cac5b4dfe74684861c10d815fb051c -> Peer cdfc2d79001d4a34ae6eddb06d8a0c0d (127.13.100.60:40697): Couldn't send request to peer cdfc2d79001d4a34ae6eddb06d8a0c0d. Status: Network error: Client connection negotiation failed: client connection to 127.13.100.60:40697: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260929 18:37:49.707091 20397 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 16cac5b4dfe74684861c10d815fb051c [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 30e554990e1d47fb8e2b183647f95d5d (127.13.100.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "16cac5b4dfe74684861c10d815fb051c" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 34955 } } peers { permanent_uuid: "30e554990e1d47fb8e2b183647f95d5d" member_type: VOTER last_known_addr { host: "127.13.100.61" port: 46389 } attrs { promote: false } } peers { permanent_uuid: "cdfc2d79001d4a34ae6eddb06d8a0c0d" member_type: NON_VOTER last_known_addr { host: "127.13.100.60" port: 40697 } attrs { promote: true } } }
I20260929 18:37:49.707485 20448 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 30e554990e1d47fb8e2b183647f95d5d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 30e554990e1d47fb8e2b183647f95d5d (127.13.100.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "16cac5b4dfe74684861c10d815fb051c" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 34955 } } peers { permanent_uuid: "30e554990e1d47fb8e2b183647f95d5d" member_type: VOTER last_known_addr { host: "127.13.100.61" port: 46389 } attrs { promote: false } } peers { permanent_uuid: "cdfc2d79001d4a34ae6eddb06d8a0c0d" member_type: NON_VOTER last_known_addr { host: "127.13.100.60" port: 40697 } attrs { promote: true } } }
I20260929 18:37:49.708686 20397 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16cac5b4dfe74684861c10d815fb051c [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "16cac5b4dfe74684861c10d815fb051c" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "16cac5b4dfe74684861c10d815fb051c" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 34955 } } peers { permanent_uuid: "30e554990e1d47fb8e2b183647f95d5d" member_type: VOTER last_known_addr { host: "127.13.100.61" port: 46389 } attrs { promote: false } } peers { permanent_uuid: "cdfc2d79001d4a34ae6eddb06d8a0c0d" member_type: NON_VOTER last_known_addr { host: "127.13.100.60" port: 40697 } attrs { promote: true } } }
I20260929 18:37:49.708801 20397 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16cac5b4dfe74684861c10d815fb051c [sys.catalog]: This master's current role is: LEADER
I20260929 18:37:49.708844 20396 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16cac5b4dfe74684861c10d815fb051c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "16cac5b4dfe74684861c10d815fb051c" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "16cac5b4dfe74684861c10d815fb051c" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 34955 } } peers { permanent_uuid: "30e554990e1d47fb8e2b183647f95d5d" member_type: VOTER last_known_addr { host: "127.13.100.61" port: 46389 } attrs { promote: false } } peers { permanent_uuid: "cdfc2d79001d4a34ae6eddb06d8a0c0d" member_type: NON_VOTER last_known_addr { host: "127.13.100.60" port: 40697 } attrs { promote: true } } }
I20260929 18:37:49.708947 20396 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16cac5b4dfe74684861c10d815fb051c [sys.catalog]: This master's current role is: LEADER
I20260929 18:37:49.709549 20585 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30e554990e1d47fb8e2b183647f95d5d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "16cac5b4dfe74684861c10d815fb051c" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "16cac5b4dfe74684861c10d815fb051c" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 34955 } } peers { permanent_uuid: "30e554990e1d47fb8e2b183647f95d5d" member_type: VOTER last_known_addr { host: "127.13.100.61" port: 46389 } attrs { promote: false } } peers { permanent_uuid: "cdfc2d79001d4a34ae6eddb06d8a0c0d" member_type: NON_VOTER last_known_addr { host: "127.13.100.60" port: 40697 } attrs { promote: true } } }
I20260929 18:37:49.709663 20585 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30e554990e1d47fb8e2b183647f95d5d [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:37:49.711877 20410 master.cc:552] Master@127.13.100.60:40697 shutdown complete.
I20260929 18:37:49.711930 20410 master_runner.cc:305] Clearing existing system tablet
I20260929 18:37:49.712170 20410 fs_manager.cc:359] Metadata directory not provided
I20260929 18:37:49.712224 20410 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:37:49.713515 20410 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 18:37:49.714419 20587 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:37:49.715591 20410 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:37:49.715720 20410 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.000s
I20260929 18:37:49.715754 20410 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/master-2/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/master-2/wal
uuid: "cdfc2d79001d4a34ae6eddb06d8a0c0d"
format_stamp: "Formatted at 2026-09-29 18:37:49 on dist-test-slave-j1xs"
server_key: "38298185906dff45f48c4954a86215c3"
server_key_iv: "a84c6fe361ac8c1e10f5e9783ed47e4f"
server_key_version: "encryptionkey@0"
I20260929 18:37:49.717669 20410 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P cdfc2d79001d4a34ae6eddb06d8a0c0d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:37:49.720012 20410 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P cdfc2d79001d4a34ae6eddb06d8a0c0d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260929 18:37:49.720098 20410 log.cc:1196] T 00000000000000000000000000000000 P cdfc2d79001d4a34ae6eddb06d8a0c0d: Deleting WAL directory at /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260929 18:37:49.720342 20410 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P cdfc2d79001d4a34ae6eddb06d8a0c0d: Deleting consensus metadata
I20260929 18:37:49.720497 20410 master_runner.cc:315] Copying system tablet from 127.13.100.62:34955
I20260929 18:37:49.721568 20410 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P cdfc2d79001d4a34ae6eddb06d8a0c0d: tablet copy: Beginning tablet copy session from remote peer at address 127.13.100.62:34955
I20260929 18:37:49.731671 19876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:49.722535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51544 (local address 127.13.100.62:34955)
0929 18:37:49.722659 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:49.722665 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:49.722682 (+ 17us) server_negotiation.cc:407] Connection header received
0929 18:37:49.722732 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:49.722735 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:49.722787 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:49.722889 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:49.723388 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:49.724197 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:49.725441 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:49.725783 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:49.725855 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:49.727297 (+ 1442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:49.727302 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:49.727305 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:37:49.727335 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:49.729758 (+ 2423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:49.730276 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:49.730280 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:49.730283 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:49.730385 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:49.730684 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:49.730688 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:49.730691 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:49.730976 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:49.731094 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:49.731378 (+ 284us) server_negotiation.cc:299] Negotiation successful
0929 18:37:49.731501 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 18:37:49.731798 20593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:49.722131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:34955 (local address 127.0.0.1:51544)
0929 18:37:49.722417 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:49.722433 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:49.722538 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:49.722915 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:49.722918 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:49.723128 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:37:49.723248 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:49.723255 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:49.724344 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:49.724347 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:49.725300 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:49.725311 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:49.726044 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:49.726047 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:49.726130 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:49.726712 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:37:49.726727 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:37:49.727131 (+ 404us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:37:49.729880 (+ 2749us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:49.729883 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:49.729907 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:49.730166 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:49.730484 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:49.730488 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:49.730490 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:49.730581 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:49.731086 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:37:49.731089 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:37:49.731193 (+ 104us) client_negotiation.cc:769] Sending connection context
0929 18:37:49.731296 (+ 103us) client_negotiation.cc:240] Negotiation successful
0929 18:37:49.731432 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":126,"threads_started":1}
I20260929 18:37:49.732549 19857 tablet_copy_service.cc:140] P 16cac5b4dfe74684861c10d815fb051c: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer cdfc2d79001d4a34ae6eddb06d8a0c0d ({username='oryx', principal='oryx/127.13.100.60@KRBTEST.COM'} at 127.0.0.1:51544)
I20260929 18:37:49.732654 19857 tablet_copy_service.cc:161] P 16cac5b4dfe74684861c10d815fb051c: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer cdfc2d79001d4a34ae6eddb06d8a0c0d at {username='oryx', principal='oryx/127.13.100.60@KRBTEST.COM'} at 127.0.0.1:51544: session id = cdfc2d79001d4a34ae6eddb06d8a0c0d-00000000000000000000000000000000
I20260929 18:37:49.733768 19857 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 16cac5b4dfe74684861c10d815fb051c: Tablet Copy: opened 0 blocks and 1 log segments
I20260929 18:37:49.734992 20410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260929 18:37:49.737322 20410 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P cdfc2d79001d4a34ae6eddb06d8a0c0d: tablet copy: Starting download of 0 data blocks...
I20260929 18:37:49.737542 20410 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P cdfc2d79001d4a34ae6eddb06d8a0c0d: tablet copy: Starting download of 1 WAL segments...
I20260929 18:37:49.739157 20410 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P cdfc2d79001d4a34ae6eddb06d8a0c0d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260929 18:37:49.741204 19857 tablet_copy_service.cc:342] P 16cac5b4dfe74684861c10d815fb051c: Request end of tablet copy session cdfc2d79001d4a34ae6eddb06d8a0c0d-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.13.100.60@KRBTEST.COM'} at 127.0.0.1:51544
I20260929 18:37:49.741286 19857 tablet_copy_service.cc:433] P 16cac5b4dfe74684861c10d815fb051c: ending tablet copy session cdfc2d79001d4a34ae6eddb06d8a0c0d-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer cdfc2d79001d4a34ae6eddb06d8a0c0d
I20260929 18:37:49.748368 20410 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:37:49.750121 20594 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:37:49.750331 20595 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:37:49.750497 20597 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:37:49.751355 20410 server_base.cc:1034] running on GCE node
I20260929 18:37:49.751509 20410 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:37:49.751804 20410 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:37:49.752970 20410 hybrid_clock.cc:648] HybridClock initialized: now 1790707069752947 us; error 50 us; skew 500 ppm
Sep 29 18:37:49 dist-test-slave-j1xs krb5kdc[19792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707069, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.100.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:37:49.754642 20410 init.cc:377] Logged in from keytab as oryx/127.13.100.60@KRBTEST.COM (short username oryx)
I20260929 18:37:49.755098 20410 webserver.cc:460] Webserver started at http://127.13.100.60:38681/ using document root <none> and password file <none>
I20260929 18:37:49.755287 20410 fs_manager.cc:359] Metadata directory not provided
I20260929 18:37:49.755329 20410 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:37:49.756668 20410 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260929 18:37:49.757700 20604 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:37:49.758031 20410 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 18:37:49.758108 20410 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/master-2/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/master-2/wal
uuid: "cdfc2d79001d4a34ae6eddb06d8a0c0d"
format_stamp: "Formatted at 2026-09-29 18:37:49 on dist-test-slave-j1xs"
server_key: "38298185906dff45f48c4954a86215c3"
server_key_iv: "a84c6fe361ac8c1e10f5e9783ed47e4f"
server_key_version: "encryptionkey@0"
I20260929 18:37:49.758191 20410 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:37:49.780799 20410 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:37:49.783069 20410 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:37:49.788105 20410 rpc_server.cc:307] RPC server started. Bound to: 127.13.100.60:40697
I20260929 18:37:49.788136 20656 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.100.60:40697 every 8 connection(s)
I20260929 18:37:49.788709 20410 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/master-2/data/info.pb
I20260929 18:37:49.790539 20657 sys_catalog.cc:263] Verifying existing consensus state
I20260929 18:37:49.791617 20657 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cdfc2d79001d4a34ae6eddb06d8a0c0d: Bootstrap starting.
I20260929 18:37:49.803241 20657 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cdfc2d79001d4a34ae6eddb06d8a0c0d: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 18:37:49.803525 20657 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cdfc2d79001d4a34ae6eddb06d8a0c0d: Bootstrap complete.
I20260929 18:37:49.804031 20657 raft_consensus.cc:348] T 00000000000000000000000000000000 P cdfc2d79001d4a34ae6eddb06d8a0c0d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "16cac5b4dfe74684861c10d815fb051c" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 34955 } } peers { permanent_uuid: "30e554990e1d47fb8e2b183647f95d5d" member_type: VOTER last_known_addr { host: "127.13.100.61" port: 46389 } attrs { promote: false } } peers { permanent_uuid: "cdfc2d79001d4a34ae6eddb06d8a0c0d" member_type: NON_VOTER last_known_addr { host: "127.13.100.60" port: 40697 } attrs { promote: true } }
I20260929 18:37:49.804172 20657 raft_consensus.cc:729] T 00000000000000000000000000000000 P cdfc2d79001d4a34ae6eddb06d8a0c0d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: cdfc2d79001d4a34ae6eddb06d8a0c0d, State: Initialized, Role: LEARNER
I20260929 18:37:49.804276 20657 consensus_queue.cc:260] T 00000000000000000000000000000000 P cdfc2d79001d4a34ae6eddb06d8a0c0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "16cac5b4dfe74684861c10d815fb051c" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 34955 } } peers { permanent_uuid: "30e554990e1d47fb8e2b183647f95d5d" member_type: VOTER last_known_addr { host: "127.13.100.61" port: 46389 } attrs { promote: false } } peers { permanent_uuid: "cdfc2d79001d4a34ae6eddb06d8a0c0d" member_type: NON_VOTER last_known_addr { host: "127.13.100.60" port: 40697 } attrs { promote: true } }
I20260929 18:37:49.804652 20663 sys_catalog.cc:455] T 00000000000000000000000000000000 P cdfc2d79001d4a34ae6eddb06d8a0c0d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "16cac5b4dfe74684861c10d815fb051c" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 34955 } } peers { permanent_uuid: "30e554990e1d47fb8e2b183647f95d5d" member_type: VOTER last_known_addr { host: "127.13.100.61" port: 46389 } attrs { promote: false } } peers { permanent_uuid: "cdfc2d79001d4a34ae6eddb06d8a0c0d" member_type: NON_VOTER last_known_addr { host: "127.13.100.60" port: 40697 } attrs { promote: true } } }
I20260929 18:37:49.804740 20657 sys_catalog.cc:564] T 00000000000000000000000000000000 P cdfc2d79001d4a34ae6eddb06d8a0c0d [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:37:49.804862 20663 sys_catalog.cc:458] T 00000000000000000000000000000000 P cdfc2d79001d4a34ae6eddb06d8a0c0d [sys.catalog]: This master's current role is: LEARNER
W20260929 18:37:49.805166 20583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:37:49.805480 20583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:37:49.805584 20583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260929 18:37:49.808789 20674 catalog_manager.cc:1259] Loaded cluster ID: 3dd0e53705814f59aa70862dfbe1aea6
I20260929 18:37:49.808849 20674 catalog_manager.cc:1552] T 00000000000000000000000000000000 P cdfc2d79001d4a34ae6eddb06d8a0c0d: loading cluster ID for follower catalog manager: success
I20260929 18:37:49.811043 20674 catalog_manager.cc:1574] T 00000000000000000000000000000000 P cdfc2d79001d4a34ae6eddb06d8a0c0d: acquiring CA information for follower catalog manager: success
W20260929 18:37:49.811060 20583 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:37:49.811139 20583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:37:49.811246 20583 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.100.1
I20260929 18:37:49.812567 20674 catalog_manager.cc:1602] T 00000000000000000000000000000000 P cdfc2d79001d4a34ae6eddb06d8a0c0d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 18:37:49.816716 20583 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.100.20:45575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/krb5kdc/oryx_127.13.100.1.keytab
--principal=oryx/127.13.100.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.100.1:38831
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.100.1
--webserver_port=46753
--tserver_master_addrs=127.13.100.62:34955,127.13.100.61:46389,127.13.100.60:40697
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20583
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.100.1
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:37:49.817761 20583 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:37:49.818902 20583 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:37:49.828733 20679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:37:49.829002 20583 server_base.cc:1034] running on GCE node
W20260929 18:37:49.828733 20676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:37:49.828733 20677 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:37:49.829577 20583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:37:49.830283 20583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:37:49.831480 20583 hybrid_clock.cc:648] HybridClock initialized: now 1790707069831468 us; error 40 us; skew 500 ppm
Sep 29 18:37:49 dist-test-slave-j1xs krb5kdc[19792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707069, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.100.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:37:49.834383 20583 init.cc:377] Logged in from keytab as oryx/127.13.100.1@KRBTEST.COM (short username oryx)
I20260929 18:37:49.835342 20583 webserver.cc:460] Webserver started at http://127.13.100.1:46753/ using document root <none> and password file <none>
I20260929 18:37:49.835917 20583 fs_manager.cc:359] Metadata directory not provided
I20260929 18:37:49.836030 20583 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:37:49.839885 20583 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260929 18:37:49.842049 20686 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:37:49.843276 20583 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:37:49.843430 20583 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/ts-0/wal
uuid: "8600dd95d7d441f9aaf8fc399cfccb14"
format_stamp: "Formatted at 2026-09-29 18:37:48 on dist-test-slave-j1xs"
server_key: "b7611c10c7dc7613975cee612df6b171"
server_key_iv: "59383d66760491e868fd84a5dc68ece3"
server_key_version: "encryptionkey@0"
I20260929 18:37:49.843979 20583 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:37:49.848922 20162 heartbeater.cc:503] Master 127.13.100.62:34955 was elected leader, sending a full tablet report...
I20260929 18:37:49.860271 20583 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:37:49.863484 20583 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:37:49.863654 20583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:37:49.864205 20583 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:37:49.865023 20583 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:37:49.865070 20583 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:37:49.865111 20583 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:37:49.865177 20583 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:37:49.874945 20583 rpc_server.cc:307] RPC server started. Bound to: 127.13.100.1:38831
I20260929 18:37:49.874963 20799 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.100.1:38831 every 8 connection(s)
I20260929 18:37:49.875938 20583 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/ts-0/data/info.pb
I20260929 18:37:49.882488 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 20583
I20260929 18:37:49.882681 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 20033
Sep 29 18:37:49 dist-test-slave-j1xs krb5kdc[19792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707069, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.100.1@KRBTEST.COM for oryx/127.13.100.61@KRBTEST.COM
Sep 29 18:37:49 dist-test-slave-j1xs krb5kdc[19792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707069, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.100.1@KRBTEST.COM for oryx/127.13.100.62@KRBTEST.COM
Sep 29 18:37:49 dist-test-slave-j1xs krb5kdc[19792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707069, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.100.1@KRBTEST.COM for oryx/127.13.100.60@KRBTEST.COM
I20260929 18:37:49.890117 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.100.2:45533
--local_ip_for_outbound_sockets=127.13.100.2
--tserver_master_addrs=127.13.100.62:34955
--webserver_port=32899
--webserver_interface=127.13.100.2
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/krb5kdc/oryx_127.13.100.2.keytab
--principal=oryx/127.13.100.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.100.20:45575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.13.100.62:34955,127.13.100.61:46389
--tserver_master_addrs=127.13.100.62:34955,127.13.100.61:46389,127.13.100.60:40697 with env {KRB5CCNAME=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 18:37:49.893818 20560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:49.877683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.1:55871 (local address 127.13.100.61:46389)
0929 18:37:49.877822 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:49.877827 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:49.879099 (+ 1272us) server_negotiation.cc:407] Connection header received
0929 18:37:49.880256 (+ 1157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:49.880260 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:49.880323 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:49.880375 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:49.881932 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:49.882734 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:49.884541 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:49.884883 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:49.884958 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:49.889916 (+ 4958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:49.889921 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:49.889924 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:37:49.889950 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:49.891578 (+ 1628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:49.892064 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:49.892067 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:49.892069 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:49.892148 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:49.892483 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:49.892486 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:49.892487 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:49.892686 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:49.892761 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:49.893531 (+ 770us) server_negotiation.cc:299] Negotiation successful
0929 18:37:49.893650 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 18:37:49.894681 19876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:49.878599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.1:49133 (local address 127.13.100.62:34955)
0929 18:37:49.878722 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:49.878727 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:49.879393 (+ 666us) server_negotiation.cc:407] Connection header received
0929 18:37:49.880234 (+ 841us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:49.880237 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:49.880295 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:49.880406 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:49.881881 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:49.882582 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:49.884546 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:49.884984 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:49.885072 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:49.889108 (+ 4036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:49.889112 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:49.889115 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:37:49.889146 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:49.892245 (+ 3099us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:49.892792 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:49.892796 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:49.892800 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:49.892895 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:49.893217 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:49.893223 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:49.893228 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:49.893467 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:49.893543 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:49.894300 (+ 757us) server_negotiation.cc:299] Negotiation successful
0929 18:37:49.894432 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 18:37:49.894903 20805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:49.878232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.61:46389 (local address 127.13.100.1:55871)
0929 18:37:49.878950 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:49.878999 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:49.880058 (+ 1059us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:49.880575 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:49.880582 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:49.881153 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:37:49.881745 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:49.881755 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:49.882900 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:49.882902 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:49.884399 (+ 1497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:49.884413 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:49.885163 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:49.885166 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:49.885359 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:49.886188 (+ 829us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:37:49.886209 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:37:49.889723 (+ 3514us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:37:49.891689 (+ 1966us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:49.891694 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:49.891699 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:49.891961 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:49.892254 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:49.892256 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:49.892258 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:49.892364 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:49.892844 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:37:49.892851 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:37:49.893239 (+ 388us) client_negotiation.cc:769] Sending connection context
0929 18:37:49.894042 (+ 803us) client_negotiation.cc:240] Negotiation successful
0929 18:37:49.894168 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"mutex_wait_us":449,"spinlock_wait_cycles":31616,"thread_start_us":686,"threads_started":1}
I20260929 18:37:49.895188 20806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:49.878232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.60:40697 (local address 127.13.100.1:48145)
0929 18:37:49.879223 (+ 991us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:49.879234 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:49.880058 (+ 824us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:49.880578 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:49.880582 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:49.881152 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:37:49.881748 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:49.881755 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:49.884934 (+ 3179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:49.884937 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:49.886227 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:49.886237 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:49.886981 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:49.886985 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:49.887082 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:49.887606 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:37:49.887620 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:37:49.889301 (+ 1681us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:37:49.891471 (+ 2170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:49.891478 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:49.891496 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:49.891864 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:49.892190 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:49.892193 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:49.892195 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:49.892370 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:49.892853 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:37:49.892855 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:37:49.893239 (+ 384us) client_negotiation.cc:769] Sending connection context
0929 18:37:49.893456 (+ 217us) client_negotiation.cc:240] Negotiation successful
0929 18:37:49.893801 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":910,"mutex_wait_us":145,"spinlock_wait_cycles":67584,"thread_start_us":171,"threads_started":1}
I20260929 18:37:49.895736 20804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:49.877957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.1:48145 (local address 127.13.100.60:40697)
0929 18:37:49.878197 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:49.878200 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:49.879325 (+ 1125us) server_negotiation.cc:407] Connection header received
0929 18:37:49.880336 (+ 1011us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:49.880339 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:49.880394 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:49.880469 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:49.881913 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:49.884813 (+ 2900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:49.886370 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:49.886733 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:49.886849 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:49.889462 (+ 2613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:49.889478 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:49.889480 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:37:49.889512 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:49.891332 (+ 1820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:49.891990 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:49.891997 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:49.892002 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:49.892088 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:49.892469 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:49.892472 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:49.892474 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:49.892735 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:49.892818 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:49.895443 (+ 2625us) server_negotiation.cc:299] Negotiation successful
0929 18:37:49.895571 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":88,"threads_started":1}
I20260929 18:37:49.896206 20802 heartbeater.cc:347] Connected to a master server at 127.13.100.61:46389
I20260929 18:37:49.896452 20802 heartbeater.cc:464] Registering TS with master...
I20260929 18:37:49.896724 20800 heartbeater.cc:347] Connected to a master server at 127.13.100.60:40697
I20260929 18:37:49.896803 20800 heartbeater.cc:464] Registering TS with master...
I20260929 18:37:49.896886 20807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:49.878466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:34955 (local address 127.13.100.1:49133)
0929 18:37:49.879292 (+ 826us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:49.879304 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:49.880035 (+ 731us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:49.880574 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:49.880581 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:49.881144 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:37:49.881736 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:49.881749 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:49.882762 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:49.882769 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:49.884402 (+ 1633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:49.884416 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:49.885244 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:49.885251 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:49.885383 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:49.886497 (+ 1114us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:37:49.886517 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:37:49.888883 (+ 2366us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:37:49.892383 (+ 3500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:49.892387 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:49.892392 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:49.892667 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:49.893014 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:49.893019 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:49.893024 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:49.893115 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:49.893631 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:37:49.893636 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:37:49.893803 (+ 167us) client_negotiation.cc:769] Sending connection context
0929 18:37:49.895425 (+ 1622us) client_negotiation.cc:240] Negotiation successful
0929 18:37:49.895613 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":768,"mutex_wait_us":213,"spinlock_wait_cycles":18688,"thread_start_us":94,"threads_started":1}
I20260929 18:37:49.897017 20802 heartbeater.cc:511] Master 127.13.100.61:46389 requested a full tablet report, sending...
I20260929 18:37:49.897017 20800 heartbeater.cc:511] Master 127.13.100.60:40697 requested a full tablet report, sending...
I20260929 18:37:49.897811 20801 heartbeater.cc:347] Connected to a master server at 127.13.100.62:34955
I20260929 18:37:49.897881 20801 heartbeater.cc:464] Registering TS with master...
I20260929 18:37:49.898082 20801 heartbeater.cc:511] Master 127.13.100.62:34955 requested a full tablet report, sending...
I20260929 18:37:49.898895 20438 ts_manager.cc:194] Registered new tserver with Master: 8600dd95d7d441f9aaf8fc399cfccb14 (127.13.100.1:38831)
I20260929 18:37:49.899097 20621 ts_manager.cc:194] Registered new tserver with Master: 8600dd95d7d441f9aaf8fc399cfccb14 (127.13.100.1:38831)
I20260929 18:37:49.899811 19837 ts_manager.cc:194] Re-registered known tserver with Master: 8600dd95d7d441f9aaf8fc399cfccb14 (127.13.100.1:38831)
I20260929 18:37:49.900415 19837 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.13.100.1@KRBTEST.COM'} at 127.13.100.1:49133
W20260929 18:37:50.004509 20808 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:37:50.004810 20808 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:37:50.004877 20808 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:37:50.009608 20808 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:37:50.009682 20808 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:37:50.009780 20808 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.100.2
I20260929 18:37:50.015167 20808 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.100.20:45575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/krb5kdc/oryx_127.13.100.2.keytab
--principal=oryx/127.13.100.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.100.2:45533
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.100.2
--webserver_port=32899
--tserver_master_addrs=127.13.100.62:34955,127.13.100.61:46389,127.13.100.60:40697
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20808
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.100.2
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:37:50.016319 20808 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:37:50.017369 20808 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:37:50.027577 20814 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:37:50.027590 20813 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:37:50.027601 20816 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:37:50.027927 20808 server_base.cc:1034] running on GCE node
I20260929 18:37:50.028386 20808 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:37:50.029060 20808 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:37:50.030243 20808 hybrid_clock.cc:648] HybridClock initialized: now 1790707070030229 us; error 31 us; skew 500 ppm
Sep 29 18:37:50 dist-test-slave-j1xs krb5kdc[19792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707070, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.100.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:37:50.032922 20808 init.cc:377] Logged in from keytab as oryx/127.13.100.2@KRBTEST.COM (short username oryx)
I20260929 18:37:50.033818 20808 webserver.cc:460] Webserver started at http://127.13.100.2:32899/ using document root <none> and password file <none>
I20260929 18:37:50.034508 20808 fs_manager.cc:359] Metadata directory not provided
I20260929 18:37:50.034591 20808 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:37:50.038430 20808 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260929 18:37:50.040357 20823 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:37:50.041633 20808 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:37:50.041767 20808 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/ts-1/wal
uuid: "a579500ca64f489886008b02eba510bf"
format_stamp: "Formatted at 2026-09-29 18:37:48 on dist-test-slave-j1xs"
server_key: "7dca658eda06cb9ea771ca699bc6c1fd"
server_key_iv: "6722dcfdfae663ac7e9c6c40b3bb39f1"
server_key_version: "encryptionkey@0"
I20260929 18:37:50.042351 20808 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:37:50.053593 20808 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:37:50.056552 20808 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:37:50.056754 20808 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:37:50.057345 20808 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:37:50.058318 20808 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:37:50.058421 20808 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:37:50.058485 20808 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:37:50.058516 20808 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:37:50.067575 20808 rpc_server.cc:307] RPC server started. Bound to: 127.13.100.2:45533
I20260929 18:37:50.067612 20936 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.100.2:45533 every 8 connection(s)
I20260929 18:37:50.068516 20808 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/ts-1/data/info.pb
I20260929 18:37:50.078058 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 20808
I20260929 18:37:50.078285 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 20167
Sep 29 18:37:50 dist-test-slave-j1xs krb5kdc[19792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707070, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.100.2@KRBTEST.COM for oryx/127.13.100.62@KRBTEST.COM
Sep 29 18:37:50 dist-test-slave-j1xs krb5kdc[19792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707070, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.100.2@KRBTEST.COM for oryx/127.13.100.60@KRBTEST.COM
Sep 29 18:37:50 dist-test-slave-j1xs krb5kdc[19792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707070, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.100.2@KRBTEST.COM for oryx/127.13.100.61@KRBTEST.COM
I20260929 18:37:50.085124 19876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:50.070948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.2:59723 (local address 127.13.100.62:34955)
0929 18:37:50.071098 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:50.071103 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:50.071779 (+ 676us) server_negotiation.cc:407] Connection header received
0929 18:37:50.073079 (+ 1300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:50.073082 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:50.073127 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:50.073221 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:50.074500 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:50.075027 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:50.076675 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:50.077011 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:50.077095 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:50.080793 (+ 3698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:50.080802 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:50.080807 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:37:50.080838 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:50.083107 (+ 2269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:50.083578 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:50.083581 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:50.083583 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:50.083669 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:50.083971 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:50.083974 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:50.083976 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:50.084188 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:50.084252 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:50.084829 (+ 577us) server_negotiation.cc:299] Negotiation successful
0929 18:37:50.084953 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 18:37:50.085129 20804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:50.071266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.2:48809 (local address 127.13.100.60:40697)
0929 18:37:50.071382 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:50.071386 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:50.072277 (+ 891us) server_negotiation.cc:407] Connection header received
0929 18:37:50.072937 (+ 660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:50.072941 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:50.072996 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:50.073102 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:50.074524 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:50.075327 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:50.076651 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:50.076885 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:50.076991 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:50.080506 (+ 3515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:50.080511 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:50.080514 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:37:50.080542 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:50.082787 (+ 2245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:50.083325 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:50.083329 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:50.083330 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:50.083410 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:50.083803 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:50.083806 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:50.083808 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:50.084039 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:50.084121 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:50.084829 (+ 708us) server_negotiation.cc:299] Negotiation successful
0929 18:37:50.084953 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 18:37:50.086292 20560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:50.070988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.2:51049 (local address 127.13.100.61:46389)
0929 18:37:50.071152 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:50.071156 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:50.071898 (+ 742us) server_negotiation.cc:407] Connection header received
0929 18:37:50.073345 (+ 1447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:50.073349 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:50.073400 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:50.073490 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:50.074525 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:50.075317 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:50.077193 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:50.077551 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:50.077685 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:50.080844 (+ 3159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:50.080851 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:50.080857 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:37:50.080889 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:50.083984 (+ 3095us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:50.084510 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:50.084516 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:50.084532 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:50.084622 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:50.084932 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:50.084938 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:50.084942 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:50.085184 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:50.085259 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:50.085918 (+ 659us) server_negotiation.cc:299] Negotiation successful
0929 18:37:50.086054 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260929 18:37:50.086828 20941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:50.070831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:34955 (local address 127.13.100.2:59723)
0929 18:37:50.071618 (+ 787us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:50.071667 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:50.072623 (+ 956us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:50.073298 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:50.073305 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:50.073814 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:37:50.074361 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:50.074374 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:50.075160 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:50.075163 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:50.076548 (+ 1385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:50.076557 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:50.077248 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:50.077252 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:50.077342 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:50.077998 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:37:50.078015 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:37:50.080637 (+ 2622us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:37:50.083216 (+ 2579us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:50.083219 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:50.083223 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:50.083467 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:50.083788 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:50.083791 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:50.083792 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:50.083872 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:50.084314 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:37:50.084316 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:37:50.084524 (+ 208us) client_negotiation.cc:769] Sending connection context
0929 18:37:50.085434 (+ 910us) client_negotiation.cc:240] Negotiation successful
0929 18:37:50.085738 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":609,"mutex_wait_us":211,"spinlock_wait_cycles":10880,"thread_start_us":119,"threads_started":1}
I20260929 18:37:50.087107 20942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:50.070831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.60:40697 (local address 127.13.100.2:48809)
0929 18:37:50.072180 (+ 1349us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:50.072193 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:50.072640 (+ 447us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:50.073296 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:50.073302 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:50.073854 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:37:50.074379 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:50.074386 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:50.075429 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:50.075432 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:50.076531 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:50.076540 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:50.077033 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:50.077037 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:50.077239 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:50.077819 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:37:50.077847 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:37:50.080325 (+ 2478us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:37:50.082904 (+ 2579us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:50.082909 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:50.082924 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:50.083206 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:50.083524 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:50.083528 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:50.083532 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:50.083693 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:50.084200 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:37:50.084207 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:37:50.084524 (+ 317us) client_negotiation.cc:769] Sending connection context
0929 18:37:50.085281 (+ 757us) client_negotiation.cc:240] Negotiation successful
0929 18:37:50.085692 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1279,"mutex_wait_us":464,"spinlock_wait_cycles":51968,"thread_start_us":102,"threads_started":1}
I20260929 18:37:50.087213 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.100.3:35631
--local_ip_for_outbound_sockets=127.13.100.3
--tserver_master_addrs=127.13.100.62:34955
--webserver_port=34909
--webserver_interface=127.13.100.3
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/krb5kdc/oryx_127.13.100.3.keytab
--principal=oryx/127.13.100.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.100.20:45575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.13.100.62:34955,127.13.100.61:46389
--tserver_master_addrs=127.13.100.62:34955,127.13.100.61:46389,127.13.100.60:40697 with env {KRB5CCNAME=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 18:37:50.088315 20938 heartbeater.cc:347] Connected to a master server at 127.13.100.62:34955
I20260929 18:37:50.088335 20943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:50.070831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.61:46389 (local address 127.13.100.2:51049)
0929 18:37:50.071794 (+ 963us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:50.071806 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:50.072623 (+ 817us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:50.073523 (+ 900us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:50.073526 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:50.073844 (+ 318us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:37:50.074377 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:50.074386 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:50.076212 (+ 1826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:50.076215 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:50.077005 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:50.077012 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:50.077691 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:50.077695 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:50.077781 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:50.078329 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:37:50.078349 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:37:50.080686 (+ 2337us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:37:50.084112 (+ 3426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:50.084118 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:50.084123 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:50.084379 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:50.084738 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:50.084744 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:50.084749 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:50.084836 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:50.085598 (+ 762us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:37:50.085604 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:37:50.085710 (+ 106us) client_negotiation.cc:769] Sending connection context
0929 18:37:50.086563 (+ 853us) client_negotiation.cc:240] Negotiation successful
0929 18:37:50.086726 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":893,"mutex_wait_us":139,"spinlock_wait_cycles":23808,"thread_start_us":149,"threads_started":1}
I20260929 18:37:50.088609 20938 heartbeater.cc:464] Registering TS with master...
I20260929 18:37:50.088812 20937 heartbeater.cc:347] Connected to a master server at 127.13.100.60:40697
I20260929 18:37:50.088905 20937 heartbeater.cc:464] Registering TS with master...
I20260929 18:37:50.089319 20937 heartbeater.cc:511] Master 127.13.100.60:40697 requested a full tablet report, sending...
I20260929 18:37:50.089319 20938 heartbeater.cc:511] Master 127.13.100.62:34955 requested a full tablet report, sending...
I20260929 18:37:50.089319 20939 heartbeater.cc:347] Connected to a master server at 127.13.100.61:46389
I20260929 18:37:50.089474 20939 heartbeater.cc:464] Registering TS with master...
I20260929 18:37:50.089618 20939 heartbeater.cc:511] Master 127.13.100.61:46389 requested a full tablet report, sending...
I20260929 18:37:50.090505 19837 ts_manager.cc:194] Re-registered known tserver with Master: a579500ca64f489886008b02eba510bf (127.13.100.2:45533)
I20260929 18:37:50.090682 20438 ts_manager.cc:194] Registered new tserver with Master: a579500ca64f489886008b02eba510bf (127.13.100.2:45533)
I20260929 18:37:50.091284 19837 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.13.100.2@KRBTEST.COM'} at 127.13.100.2:59723
I20260929 18:37:50.091373 20621 ts_manager.cc:194] Registered new tserver with Master: a579500ca64f489886008b02eba510bf (127.13.100.2:45533)
W20260929 18:37:50.199662 20944 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:37:50.199955 20944 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:37:50.200001 20944 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:37:50.204142 20944 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:37:50.204203 20944 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:37:50.204263 20944 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.100.3
I20260929 18:37:50.210196 20944 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.100.20:45575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/krb5kdc/oryx_127.13.100.3.keytab
--principal=oryx/127.13.100.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.100.3:35631
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.100.3
--webserver_port=34909
--tserver_master_addrs=127.13.100.62:34955,127.13.100.61:46389,127.13.100.60:40697
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20944
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.100.3
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:37:50.211447 20944 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:37:50.212509 20944 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:37:50.222683 20949 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:37:50.222865 20944 server_base.cc:1034] running on GCE node
W20260929 18:37:50.222669 20950 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:37:50.222648 20952 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:37:50.223414 20944 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:37:50.224058 20944 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:37:50.225251 20944 hybrid_clock.cc:648] HybridClock initialized: now 1790707070225237 us; error 26 us; skew 500 ppm
Sep 29 18:37:50 dist-test-slave-j1xs krb5kdc[19792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707070, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.100.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:37:50.228127 20944 init.cc:377] Logged in from keytab as oryx/127.13.100.3@KRBTEST.COM (short username oryx)
I20260929 18:37:50.229022 20944 webserver.cc:460] Webserver started at http://127.13.100.3:34909/ using document root <none> and password file <none>
I20260929 18:37:50.229557 20944 fs_manager.cc:359] Metadata directory not provided
I20260929 18:37:50.229632 20944 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:37:50.233392 20944 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.002s
I20260929 18:37:50.236078 20959 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:37:50.237434 20944 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 18:37:50.237532 20944 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/ts-2/wal
uuid: "2fbaf12215ba453bb80f9cccaaff2e73"
format_stamp: "Formatted at 2026-09-29 18:37:49 on dist-test-slave-j1xs"
server_key: "b10c7927b6ad724fcd455be21df63d8f"
server_key_iv: "737a4e85ca88414ece4a1c49e1cbc3af"
server_key_version: "encryptionkey@0"
I20260929 18:37:50.238152 20944 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:37:50.251401 20944 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:37:50.254767 20944 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:37:50.254925 20944 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:37:50.255514 20944 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:37:50.256394 20944 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:37:50.256443 20944 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:37:50.256506 20944 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:37:50.256537 20944 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:37:50.265964 20944 rpc_server.cc:307] RPC server started. Bound to: 127.13.100.3:35631
I20260929 18:37:50.266003 21073 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.100.3:35631 every 8 connection(s)
I20260929 18:37:50.266988 20944 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/ts-2/data/info.pb
I20260929 18:37:50.275910 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 20944
Sep 29 18:37:50 dist-test-slave-j1xs krb5kdc[19792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707070, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.100.3@KRBTEST.COM for oryx/127.13.100.60@KRBTEST.COM
Sep 29 18:37:50 dist-test-slave-j1xs krb5kdc[19792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707070, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.100.3@KRBTEST.COM for oryx/127.13.100.62@KRBTEST.COM
Sep 29 18:37:50 dist-test-slave-j1xs krb5kdc[19792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707070, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.100.3@KRBTEST.COM for oryx/127.13.100.61@KRBTEST.COM
I20260929 18:37:50.284502 21079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:50.270072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.60:40697 (local address 127.13.100.3:43767)
0929 18:37:50.270434 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:50.270446 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:50.271232 (+ 786us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:50.271817 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:50.271819 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:50.272360 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:37:50.272953 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:50.272961 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:50.273746 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:50.273749 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:50.274922 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:50.274931 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:50.275648 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:50.275652 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:50.276005 (+ 353us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:50.276826 (+ 821us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:37:50.276837 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:37:50.278692 (+ 1855us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:37:50.281017 (+ 2325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:50.281026 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:50.281045 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:50.281413 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:50.281953 (+ 540us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:50.281957 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:50.281959 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:50.282128 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:50.282585 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:37:50.282592 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:37:50.282909 (+ 317us) client_negotiation.cc:769] Sending connection context
0929 18:37:50.283093 (+ 184us) client_negotiation.cc:240] Negotiation successful
0929 18:37:50.283417 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"mutex_wait_us":405,"spinlock_wait_cycles":11648,"thread_start_us":115,"threads_started":1}
I20260929 18:37:50.284997 21078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:50.269147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.61:46389 (local address 127.13.100.3:45485)
0929 18:37:50.269858 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:50.269960 (+ 102us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:50.271232 (+ 1272us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:50.271805 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:50.271814 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:50.272351 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:37:50.272941 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:50.272954 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:50.273743 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:50.273748 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:50.274920 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:50.274928 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:50.275648 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:50.275652 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:50.276006 (+ 354us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:50.276785 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:37:50.276812 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:37:50.280449 (+ 3637us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:37:50.283135 (+ 2686us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:50.283141 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:50.283143 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:50.283379 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:50.283675 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:50.283681 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:50.283683 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:50.283766 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:50.284234 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:37:50.284237 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:37:50.284320 (+ 83us) client_negotiation.cc:769] Sending connection context
0929 18:37:50.284408 (+ 88us) client_negotiation.cc:240] Negotiation successful
0929 18:37:50.284539 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"mutex_wait_us":234,"spinlock_wait_cycles":28544,"thread_start_us":111,"threads_started":1}
I20260929 18:37:50.285297 19876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:50.270202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.3:59433 (local address 127.13.100.62:34955)
0929 18:37:50.270312 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:50.270317 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:50.270744 (+ 427us) server_negotiation.cc:407] Connection header received
0929 18:37:50.271887 (+ 1143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:50.271891 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:50.271946 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:50.272041 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:50.273772 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:50.274574 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:50.275468 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:50.275775 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:50.275875 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:50.281514 (+ 5639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:50.281518 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:50.281522 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:37:50.281554 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:50.283270 (+ 1716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:50.283723 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:50.283728 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:50.283729 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:50.283807 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:50.284092 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:50.284095 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:50.284097 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:50.284293 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:50.284359 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:50.284978 (+ 619us) server_negotiation.cc:299] Negotiation successful
0929 18:37:50.285099 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 18:37:50.285523 20804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:50.270468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.3:43767 (local address 127.13.100.60:40697)
0929 18:37:50.270606 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:50.270609 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:50.270623 (+ 14us) server_negotiation.cc:407] Connection header received
0929 18:37:50.271622 (+ 999us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:50.271626 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:50.271681 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:50.271763 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:50.273111 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:50.273635 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:50.275083 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:50.275429 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:50.275528 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:50.278886 (+ 3358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:50.278890 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:50.278892 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:37:50.278919 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:50.280684 (+ 1765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:50.281739 (+ 1055us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:50.281742 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:50.281744 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:50.281825 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:50.282245 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:50.282248 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:50.282249 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:50.282447 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:50.282533 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:50.285222 (+ 2689us) server_negotiation.cc:299] Negotiation successful
0929 18:37:50.285351 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 18:37:50.285770 21080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:50.270106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:34955 (local address 127.13.100.3:59433)
0929 18:37:50.270668 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:50.270681 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:50.271254 (+ 573us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:50.272072 (+ 818us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:50.272077 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:50.272361 (+ 284us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:37:50.272956 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:50.272961 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:50.274690 (+ 1729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:50.274693 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:50.275353 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:50.275360 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:50.276084 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:50.276087 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:50.276195 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:50.277181 (+ 986us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:37:50.277196 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:37:50.281029 (+ 3833us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:37:50.283381 (+ 2352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:50.283384 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:50.283385 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:50.283618 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:50.283915 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:50.283920 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:50.283922 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:50.283990 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:50.284426 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:37:50.284428 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:37:50.284759 (+ 331us) client_negotiation.cc:769] Sending connection context
0929 18:37:50.285462 (+ 703us) client_negotiation.cc:240] Negotiation successful
0929 18:37:50.285608 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"mutex_wait_us":138,"spinlock_wait_cycles":15104,"thread_start_us":98,"threads_started":1}
I20260929 18:37:50.286684 20560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:50.268743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.3:45485 (local address 127.13.100.61:46389)
0929 18:37:50.268876 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:50.268879 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:50.270074 (+ 1195us) server_negotiation.cc:407] Connection header received
0929 18:37:50.271472 (+ 1398us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:50.271475 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:50.271531 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:50.271621 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:50.273087 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:50.273608 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:50.275042 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:50.275367 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:50.275474 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:50.280596 (+ 5122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:50.280600 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:50.280604 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:37:50.280638 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:50.283002 (+ 2364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:50.283484 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:50.283488 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:50.283490 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:50.283570 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:50.283867 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:50.283873 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:50.283877 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:50.284110 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:50.284194 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:50.286413 (+ 2219us) server_negotiation.cc:299] Negotiation successful
0929 18:37:50.286529 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 18:37:50.286659 21075 heartbeater.cc:347] Connected to a master server at 127.13.100.62:34955
I20260929 18:37:50.286911 21075 heartbeater.cc:464] Registering TS with master...
I20260929 18:37:50.287242 21076 heartbeater.cc:347] Connected to a master server at 127.13.100.61:46389
I20260929 18:37:50.287338 21076 heartbeater.cc:464] Registering TS with master...
I20260929 18:37:50.287496 21075 heartbeater.cc:511] Master 127.13.100.62:34955 requested a full tablet report, sending...
I20260929 18:37:50.287523 21076 heartbeater.cc:511] Master 127.13.100.61:46389 requested a full tablet report, sending...
I20260929 18:37:50.286664 21074 heartbeater.cc:347] Connected to a master server at 127.13.100.60:40697
I20260929 18:37:50.288143 21074 heartbeater.cc:464] Registering TS with master...
I20260929 18:37:50.288340 21074 heartbeater.cc:511] Master 127.13.100.60:40697 requested a full tablet report, sending...
I20260929 18:37:50.288717 20438 ts_manager.cc:194] Registered new tserver with Master: 2fbaf12215ba453bb80f9cccaaff2e73 (127.13.100.3:35631)
I20260929 18:37:50.288983 20621 ts_manager.cc:194] Registered new tserver with Master: 2fbaf12215ba453bb80f9cccaaff2e73 (127.13.100.3:35631)
I20260929 18:37:50.289604 21085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:50.279550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51558 (local address 127.13.100.62:34955)
0929 18:37:50.279821 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:50.279830 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:50.280296 (+ 466us) server_negotiation.cc:407] Connection header received
0929 18:37:50.280415 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:50.280419 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:50.280478 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:50.280577 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:50.281604 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:50.282412 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:50.283514 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:50.283823 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:50.283916 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:50.285558 (+ 1642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:50.285562 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:50.285564 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:37:50.285589 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:50.287492 (+ 1903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:50.287978 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:50.287981 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:50.287983 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:50.288071 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:50.288390 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:50.288394 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:50.288397 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:50.288594 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:50.288712 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:50.289321 (+ 609us) server_negotiation.cc:299] Negotiation successful
0929 18:37:50.289439 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":73,"threads_started":1}
I20260929 18:37:50.289912 19837 ts_manager.cc:194] Re-registered known tserver with Master: 2fbaf12215ba453bb80f9cccaaff2e73 (127.13.100.3:35631)
I20260929 18:37:50.290655 19837 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.13.100.3@KRBTEST.COM'} at 127.13.100.3:59433
I20260929 18:37:50.292842 19837 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51558:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260929 18:37:50.296443 20804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:50.280840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54480 (local address 127.13.100.60:40697)
0929 18:37:50.285700 (+ 4860us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:50.285706 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:50.287421 (+ 1715us) server_negotiation.cc:407] Connection header received
0929 18:37:50.287663 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:50.287666 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:50.287711 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:50.287759 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:50.288693 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:50.289414 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:50.290915 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:50.291221 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:50.291316 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:50.293032 (+ 1716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:50.293036 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:50.293038 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:37:50.293062 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:50.294705 (+ 1643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:50.295124 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:50.295128 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:50.295129 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:50.295212 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:50.295537 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:50.295541 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:50.295543 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:50.295756 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:50.295885 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:50.296186 (+ 301us) server_negotiation.cc:299] Negotiation successful
0929 18:37:50.296299 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4797,"thread_start_us":77,"threads_started":1}
W20260929 18:37:50.297353 19837 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:37:50.299286 21088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:50.280023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48340 (local address 127.13.100.61:46389)
0929 18:37:50.289469 (+ 9446us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:50.289473 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:50.289487 (+ 14us) server_negotiation.cc:407] Connection header received
0929 18:37:50.289519 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:50.289522 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:50.289562 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:50.289664 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:50.291759 (+ 2095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:50.292425 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:50.293539 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:50.293873 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:50.293994 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:50.295486 (+ 1492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:50.295493 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:50.295499 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:37:50.295528 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:50.297483 (+ 1955us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:50.297991 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:50.297995 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:50.297997 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:50.298086 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:50.298383 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:50.298386 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:50.298387 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:50.298571 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:50.298669 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:50.298952 (+ 283us) server_negotiation.cc:299] Negotiation successful
0929 18:37:50.299086 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6703,"thread_start_us":107,"threads_started":1}
I20260929 18:37:50.315279 21099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:50.309562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.1:38831 (local address 127.0.0.1:42818)
0929 18:37:50.310189 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:50.310205 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:50.310296 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:50.311307 (+ 1011us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:50.311310 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:50.311326 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:37:50.311412 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:50.311416 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:50.312782 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:50.312785 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:50.314018 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:50.314030 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:50.314878 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:50.314882 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:50.314966 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:50.314990 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 18:37:50.315046 (+ 56us) client_negotiation.cc:240] Negotiation successful
0929 18:37:50.315116 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":96,"threads_started":1}
I20260929 18:37:50.315304 21100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:50.309974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42818 (local address 127.13.100.1:38831)
0929 18:37:50.310368 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:50.310835 (+ 467us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:50.310863 (+ 28us) server_negotiation.cc:407] Connection header received
0929 18:37:50.310953 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:50.310958 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:50.311181 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:50.311309 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:37:50.311536 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:50.312663 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:50.314165 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:50.314763 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:50.314870 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:50.315013 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:50.315070 (+ 57us) server_negotiation.cc:299] Negotiation successful
0929 18:37:50.315158 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":93,"threads_started":1}
I20260929 18:37:50.316591 21096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:50.309482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.3:35631 (local address 127.0.0.1:55552)
0929 18:37:50.309931 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:50.309949 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:50.310047 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:50.312392 (+ 2345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:50.312395 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:50.312410 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:37:50.312485 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:50.312490 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:50.314083 (+ 1593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:50.314087 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:50.315179 (+ 1092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:50.315187 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:50.316214 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:50.316218 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:50.316299 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:50.316320 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 18:37:50.316383 (+ 63us) client_negotiation.cc:240] Negotiation successful
0929 18:37:50.316457 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":341,"mutex_wait_us":21,"thread_start_us":120,"threads_started":1}
I20260929 18:37:50.316635 21098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:50.309680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55552 (local address 127.13.100.3:35631)
0929 18:37:50.311859 (+ 2179us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:50.311869 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:50.311889 (+ 20us) server_negotiation.cc:407] Connection header received
0929 18:37:50.311954 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:50.311958 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:50.312132 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:50.312271 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:37:50.312593 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:50.313966 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:50.315303 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:50.316120 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:50.316224 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:50.316352 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:50.316403 (+ 51us) server_negotiation.cc:299] Negotiation successful
0929 18:37:50.316479 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2082,"thread_start_us":103,"threads_started":1}
I20260929 18:37:50.317276 21097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:50.309939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.2:45533 (local address 127.0.0.1:39440)
0929 18:37:50.311522 (+ 1583us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:50.311535 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:50.311627 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:50.312100 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:50.312103 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:50.312119 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:37:50.312198 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:50.312203 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:50.313438 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:50.313443 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:50.314593 (+ 1150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:50.314603 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:50.316911 (+ 2308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:50.316914 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:50.317009 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:50.317029 (+ 20us) client_negotiation.cc:769] Sending connection context
0929 18:37:50.317098 (+ 69us) client_negotiation.cc:240] Negotiation successful
0929 18:37:50.317162 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1496,"mutex_wait_us":19,"thread_start_us":122,"threads_started":1}
I20260929 18:37:50.317489 21101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:50.310381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39440 (local address 127.13.100.2:45533)
0929 18:37:50.311072 (+ 691us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:50.311080 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:50.311635 (+ 555us) server_negotiation.cc:407] Connection header received
0929 18:37:50.311707 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:50.311711 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:50.311950 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:50.312117 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:37:50.312299 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:50.313330 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:50.314728 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:50.316635 (+ 1907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:50.316732 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:50.316854 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:50.317148 (+ 294us) server_negotiation.cc:299] Negotiation successful
0929 18:37:50.317246 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":609,"thread_start_us":513,"threads_started":1}
I20260929 18:37:50.318390 20734 tablet_service.cc:1467] Processing CreateTablet for tablet 2c3d9d44a48742319edb4aebcbe8e71c (DEFAULT_TABLE table=test-table [id=05cb5e8a0e484dffaa4e7a1c0cc3f435]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:37:50.319453 20734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c3d9d44a48742319edb4aebcbe8e71c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:37:50.319887 21007 tablet_service.cc:1467] Processing CreateTablet for tablet 2c3d9d44a48742319edb4aebcbe8e71c (DEFAULT_TABLE table=test-table [id=05cb5e8a0e484dffaa4e7a1c0cc3f435]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:37:50.319887 20871 tablet_service.cc:1467] Processing CreateTablet for tablet 2c3d9d44a48742319edb4aebcbe8e71c (DEFAULT_TABLE table=test-table [id=05cb5e8a0e484dffaa4e7a1c0cc3f435]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:37:50.320815 20871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c3d9d44a48742319edb4aebcbe8e71c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:37:50.320815 21007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c3d9d44a48742319edb4aebcbe8e71c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:37:50.327762 20804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:50.322701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54492 (local address 127.13.100.60:40697)
0929 18:37:50.322811 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:50.322815 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:50.322829 (+ 14us) server_negotiation.cc:407] Connection header received
0929 18:37:50.322941 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:50.322944 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:50.323004 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:50.323069 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:37:50.323410 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:50.324152 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:50.326369 (+ 2217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:50.327019 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:50.327098 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:50.327237 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:50.327574 (+ 337us) server_negotiation.cc:299] Negotiation successful
0929 18:37:50.327647 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 18:37:50.328293 21097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:50.322468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.60:40697 (local address 127.0.0.1:54492)
0929 18:37:50.322714 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:50.322729 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:50.322816 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:50.323198 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:50.323201 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:50.323217 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:37:50.323293 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:50.323298 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:50.324290 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:50.324293 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:50.326232 (+ 1939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:50.326242 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:50.327330 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:50.327334 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:50.327421 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:50.327441 (+ 20us) client_negotiation.cc:769] Sending connection context
0929 18:37:50.328097 (+ 656us) client_negotiation.cc:240] Negotiation successful
0929 18:37:50.328181 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181}
I20260929 18:37:50.329332 21103 tablet_bootstrap.cc:492] T 2c3d9d44a48742319edb4aebcbe8e71c P 8600dd95d7d441f9aaf8fc399cfccb14: Bootstrap starting.
I20260929 18:37:50.329444 20631 raft_consensus.cc:1203] T 00000000000000000000000000000000 P cdfc2d79001d4a34ae6eddb06d8a0c0d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.9] Dedup: 1.7->[1.8-1.9]
I20260929 18:37:50.331723 21103 tablet_bootstrap.cc:654] T 2c3d9d44a48742319edb4aebcbe8e71c P 8600dd95d7d441f9aaf8fc399cfccb14: Neither blocks nor log segments found. Creating new log.
I20260929 18:37:50.332636 21103 log.cc:824] T 2c3d9d44a48742319edb4aebcbe8e71c P 8600dd95d7d441f9aaf8fc399cfccb14: Log is configured to *not* fsync() on all Append() calls
I20260929 18:37:50.332588 21104 sys_catalog.cc:455] T 00000000000000000000000000000000 P cdfc2d79001d4a34ae6eddb06d8a0c0d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 16cac5b4dfe74684861c10d815fb051c. Latest consensus state: current_term: 1 leader_uuid: "16cac5b4dfe74684861c10d815fb051c" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "16cac5b4dfe74684861c10d815fb051c" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 34955 } } peers { permanent_uuid: "30e554990e1d47fb8e2b183647f95d5d" member_type: VOTER last_known_addr { host: "127.13.100.61" port: 46389 } attrs { promote: false } } peers { permanent_uuid: "cdfc2d79001d4a34ae6eddb06d8a0c0d" member_type: NON_VOTER last_known_addr { host: "127.13.100.60" port: 40697 } attrs { promote: true } } }
I20260929 18:37:50.332720 21104 sys_catalog.cc:458] T 00000000000000000000000000000000 P cdfc2d79001d4a34ae6eddb06d8a0c0d [sys.catalog]: This master's current role is: LEARNER
I20260929 18:37:50.333345 21102 tablet_bootstrap.cc:492] T 2c3d9d44a48742319edb4aebcbe8e71c P a579500ca64f489886008b02eba510bf: Bootstrap starting.
I20260929 18:37:50.335565 21102 tablet_bootstrap.cc:654] T 2c3d9d44a48742319edb4aebcbe8e71c P a579500ca64f489886008b02eba510bf: Neither blocks nor log segments found. Creating new log.
I20260929 18:37:50.336324 21102 log.cc:824] T 2c3d9d44a48742319edb4aebcbe8e71c P a579500ca64f489886008b02eba510bf: Log is configured to *not* fsync() on all Append() calls
I20260929 18:37:50.338124 21103 tablet_bootstrap.cc:492] T 2c3d9d44a48742319edb4aebcbe8e71c P 8600dd95d7d441f9aaf8fc399cfccb14: No bootstrap required, opened a new log
I20260929 18:37:50.338321 21103 ts_tablet_manager.cc:1397] T 2c3d9d44a48742319edb4aebcbe8e71c P 8600dd95d7d441f9aaf8fc399cfccb14: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260929 18:37:50.338591 21102 tablet_bootstrap.cc:492] T 2c3d9d44a48742319edb4aebcbe8e71c P a579500ca64f489886008b02eba510bf: No bootstrap required, opened a new log
I20260929 18:37:50.338764 21102 ts_tablet_manager.cc:1397] T 2c3d9d44a48742319edb4aebcbe8e71c P a579500ca64f489886008b02eba510bf: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 18:37:50.341034 21111 tablet_bootstrap.cc:492] T 2c3d9d44a48742319edb4aebcbe8e71c P 2fbaf12215ba453bb80f9cccaaff2e73: Bootstrap starting.
I20260929 18:37:50.343258 21111 tablet_bootstrap.cc:654] T 2c3d9d44a48742319edb4aebcbe8e71c P 2fbaf12215ba453bb80f9cccaaff2e73: Neither blocks nor log segments found. Creating new log.
I20260929 18:37:50.343473 21103 raft_consensus.cc:348] T 2c3d9d44a48742319edb4aebcbe8e71c P 8600dd95d7d441f9aaf8fc399cfccb14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fbaf12215ba453bb80f9cccaaff2e73" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 35631 } } peers { permanent_uuid: "8600dd95d7d441f9aaf8fc399cfccb14" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 38831 } } peers { permanent_uuid: "a579500ca64f489886008b02eba510bf" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 45533 } }
I20260929 18:37:50.343473 21102 raft_consensus.cc:348] T 2c3d9d44a48742319edb4aebcbe8e71c P a579500ca64f489886008b02eba510bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fbaf12215ba453bb80f9cccaaff2e73" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 35631 } } peers { permanent_uuid: "8600dd95d7d441f9aaf8fc399cfccb14" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 38831 } } peers { permanent_uuid: "a579500ca64f489886008b02eba510bf" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 45533 } }
I20260929 18:37:50.343750 21102 raft_consensus.cc:374] T 2c3d9d44a48742319edb4aebcbe8e71c P a579500ca64f489886008b02eba510bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:37:50.343750 21103 raft_consensus.cc:374] T 2c3d9d44a48742319edb4aebcbe8e71c P 8600dd95d7d441f9aaf8fc399cfccb14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:37:50.343815 21103 raft_consensus.cc:729] T 2c3d9d44a48742319edb4aebcbe8e71c P 8600dd95d7d441f9aaf8fc399cfccb14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8600dd95d7d441f9aaf8fc399cfccb14, State: Initialized, Role: FOLLOWER
I20260929 18:37:50.343823 21102 raft_consensus.cc:729] T 2c3d9d44a48742319edb4aebcbe8e71c P a579500ca64f489886008b02eba510bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a579500ca64f489886008b02eba510bf, State: Initialized, Role: FOLLOWER
I20260929 18:37:50.344069 21111 log.cc:824] T 2c3d9d44a48742319edb4aebcbe8e71c P 2fbaf12215ba453bb80f9cccaaff2e73: Log is configured to *not* fsync() on all Append() calls
I20260929 18:37:50.344295 21102 consensus_queue.cc:260] T 2c3d9d44a48742319edb4aebcbe8e71c P a579500ca64f489886008b02eba510bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fbaf12215ba453bb80f9cccaaff2e73" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 35631 } } peers { permanent_uuid: "8600dd95d7d441f9aaf8fc399cfccb14" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 38831 } } peers { permanent_uuid: "a579500ca64f489886008b02eba510bf" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 45533 } }
I20260929 18:37:50.344295 21103 consensus_queue.cc:260] T 2c3d9d44a48742319edb4aebcbe8e71c P 8600dd95d7d441f9aaf8fc399cfccb14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fbaf12215ba453bb80f9cccaaff2e73" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 35631 } } peers { permanent_uuid: "8600dd95d7d441f9aaf8fc399cfccb14" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 38831 } } peers { permanent_uuid: "a579500ca64f489886008b02eba510bf" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 45533 } }
I20260929 18:37:50.345270 21103 ts_tablet_manager.cc:1428] T 2c3d9d44a48742319edb4aebcbe8e71c P 8600dd95d7d441f9aaf8fc399cfccb14: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260929 18:37:50.345270 21102 ts_tablet_manager.cc:1428] T 2c3d9d44a48742319edb4aebcbe8e71c P a579500ca64f489886008b02eba510bf: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 18:37:50.345588 20801 heartbeater.cc:503] Master 127.13.100.62:34955 was elected leader, sending a full tablet report...
I20260929 18:37:50.345633 20938 heartbeater.cc:503] Master 127.13.100.62:34955 was elected leader, sending a full tablet report...
I20260929 18:37:50.346200 21111 tablet_bootstrap.cc:492] T 2c3d9d44a48742319edb4aebcbe8e71c P 2fbaf12215ba453bb80f9cccaaff2e73: No bootstrap required, opened a new log
I20260929 18:37:50.346387 21111 ts_tablet_manager.cc:1397] T 2c3d9d44a48742319edb4aebcbe8e71c P 2fbaf12215ba453bb80f9cccaaff2e73: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.004s
I20260929 18:37:50.351028 21111 raft_consensus.cc:348] T 2c3d9d44a48742319edb4aebcbe8e71c P 2fbaf12215ba453bb80f9cccaaff2e73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fbaf12215ba453bb80f9cccaaff2e73" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 35631 } } peers { permanent_uuid: "8600dd95d7d441f9aaf8fc399cfccb14" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 38831 } } peers { permanent_uuid: "a579500ca64f489886008b02eba510bf" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 45533 } }
I20260929 18:37:50.351323 21111 raft_consensus.cc:374] T 2c3d9d44a48742319edb4aebcbe8e71c P 2fbaf12215ba453bb80f9cccaaff2e73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:37:50.351387 21111 raft_consensus.cc:729] T 2c3d9d44a48742319edb4aebcbe8e71c P 2fbaf12215ba453bb80f9cccaaff2e73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fbaf12215ba453bb80f9cccaaff2e73, State: Initialized, Role: FOLLOWER
I20260929 18:37:50.351847 21111 consensus_queue.cc:260] T 2c3d9d44a48742319edb4aebcbe8e71c P 2fbaf12215ba453bb80f9cccaaff2e73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fbaf12215ba453bb80f9cccaaff2e73" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 35631 } } peers { permanent_uuid: "8600dd95d7d441f9aaf8fc399cfccb14" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 38831 } } peers { permanent_uuid: "a579500ca64f489886008b02eba510bf" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 45533 } }
I20260929 18:37:50.352622 21075 heartbeater.cc:503] Master 127.13.100.62:34955 was elected leader, sending a full tablet report...
I20260929 18:37:50.352952 21111 ts_tablet_manager.cc:1428] T 2c3d9d44a48742319edb4aebcbe8e71c P 2fbaf12215ba453bb80f9cccaaff2e73: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
W20260929 18:37:50.377938 20803 tablet.cc:2369] T 2c3d9d44a48742319edb4aebcbe8e71c P 8600dd95d7d441f9aaf8fc399cfccb14: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:37:50.486176 21114 raft_consensus.cc:482] T 2c3d9d44a48742319edb4aebcbe8e71c P a579500ca64f489886008b02eba510bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:37:50.486403 21114 raft_consensus.cc:504] T 2c3d9d44a48742319edb4aebcbe8e71c P a579500ca64f489886008b02eba510bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fbaf12215ba453bb80f9cccaaff2e73" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 35631 } } peers { permanent_uuid: "8600dd95d7d441f9aaf8fc399cfccb14" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 38831 } } peers { permanent_uuid: "a579500ca64f489886008b02eba510bf" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 45533 } }
I20260929 18:37:50.487874 21114 leader_election.cc:288] T 2c3d9d44a48742319edb4aebcbe8e71c P a579500ca64f489886008b02eba510bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2fbaf12215ba453bb80f9cccaaff2e73 (127.13.100.3:35631), 8600dd95d7d441f9aaf8fc399cfccb14 (127.13.100.1:38831)
I20260929 18:37:50.492810 21100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:50.488430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.2:33307 (local address 127.13.100.1:38831)
0929 18:37:50.488576 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:50.488580 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:50.488696 (+ 116us) server_negotiation.cc:407] Connection header received
0929 18:37:50.488799 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:50.488801 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:50.488847 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:50.488926 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:37:50.489118 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:50.489880 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:50.491355 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:50.491941 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:50.492066 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:50.492117 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:50.492587 (+ 470us) server_negotiation.cc:299] Negotiation successful
0929 18:37:50.492669 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 18:37:50.492817 21098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:50.488572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.2:39517 (local address 127.13.100.3:35631)
0929 18:37:50.488732 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:50.488736 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:50.488749 (+ 13us) server_negotiation.cc:407] Connection header received
0929 18:37:50.488798 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:50.488801 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:50.488847 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:50.488927 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:37:50.489317 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:50.490044 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:50.491355 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:50.491941 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:50.492066 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:50.492117 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:50.492587 (+ 470us) server_negotiation.cc:299] Negotiation successful
0929 18:37:50.492669 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 18:37:50.492848 20943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:50.488181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.3:35631 (local address 127.13.100.2:39517)
0929 18:37:50.488330 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:50.488343 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:50.488428 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:50.489137 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:50.489138 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:50.489147 (+ 9us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:37:50.489199 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:50.489201 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:50.490143 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:50.490146 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:50.491222 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:50.491231 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:50.492068 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:50.492072 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:50.492353 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:50.492450 (+ 97us) client_negotiation.cc:769] Sending connection context
0929 18:37:50.492530 (+ 80us) client_negotiation.cc:240] Negotiation successful
0929 18:37:50.492658 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63,"mutex_wait_us":179}
I20260929 18:37:50.492848 20941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:50.488444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.1:38831 (local address 127.13.100.2:33307)
0929 18:37:50.488591 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:50.488630 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:50.488723 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:50.488940 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:50.488942 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:50.488953 (+ 11us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:37:50.489048 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:50.489052 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:50.489996 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:50.489998 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:50.491222 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:50.491231 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:50.492068 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:50.492072 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:50.492351 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:50.492450 (+ 99us) client_negotiation.cc:769] Sending connection context
0929 18:37:50.492530 (+ 80us) client_negotiation.cc:240] Negotiation successful
0929 18:37:50.492658 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78,"mutex_wait_us":134}
I20260929 18:37:50.493616 21027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c3d9d44a48742319edb4aebcbe8e71c" candidate_uuid: "a579500ca64f489886008b02eba510bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fbaf12215ba453bb80f9cccaaff2e73" is_pre_election: true
I20260929 18:37:50.493959 21027 raft_consensus.cc:2454] T 2c3d9d44a48742319edb4aebcbe8e71c P 2fbaf12215ba453bb80f9cccaaff2e73 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a579500ca64f489886008b02eba510bf in term 0.
I20260929 18:37:50.494213 20754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c3d9d44a48742319edb4aebcbe8e71c" candidate_uuid: "a579500ca64f489886008b02eba510bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8600dd95d7d441f9aaf8fc399cfccb14" is_pre_election: true
I20260929 18:37:50.494433 20824 leader_election.cc:302] T 2c3d9d44a48742319edb4aebcbe8e71c P a579500ca64f489886008b02eba510bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fbaf12215ba453bb80f9cccaaff2e73, a579500ca64f489886008b02eba510bf; no voters:
I20260929 18:37:50.494477 20754 raft_consensus.cc:2454] T 2c3d9d44a48742319edb4aebcbe8e71c P 8600dd95d7d441f9aaf8fc399cfccb14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a579500ca64f489886008b02eba510bf in term 0.
I20260929 18:37:50.494696 21114 raft_consensus.cc:2781] T 2c3d9d44a48742319edb4aebcbe8e71c P a579500ca64f489886008b02eba510bf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:37:50.494791 21114 raft_consensus.cc:482] T 2c3d9d44a48742319edb4aebcbe8e71c P a579500ca64f489886008b02eba510bf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:37:50.494848 21114 raft_consensus.cc:3037] T 2c3d9d44a48742319edb4aebcbe8e71c P a579500ca64f489886008b02eba510bf [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:37:50.495847 21114 raft_consensus.cc:504] T 2c3d9d44a48742319edb4aebcbe8e71c P a579500ca64f489886008b02eba510bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fbaf12215ba453bb80f9cccaaff2e73" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 35631 } } peers { permanent_uuid: "8600dd95d7d441f9aaf8fc399cfccb14" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 38831 } } peers { permanent_uuid: "a579500ca64f489886008b02eba510bf" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 45533 } }
I20260929 18:37:50.496184 21114 leader_election.cc:288] T 2c3d9d44a48742319edb4aebcbe8e71c P a579500ca64f489886008b02eba510bf [CANDIDATE]: Term 1 election: Requested vote from peers 2fbaf12215ba453bb80f9cccaaff2e73 (127.13.100.3:35631), 8600dd95d7d441f9aaf8fc399cfccb14 (127.13.100.1:38831)
I20260929 18:37:50.496512 21027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c3d9d44a48742319edb4aebcbe8e71c" candidate_uuid: "a579500ca64f489886008b02eba510bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fbaf12215ba453bb80f9cccaaff2e73"
I20260929 18:37:50.496512 20754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c3d9d44a48742319edb4aebcbe8e71c" candidate_uuid: "a579500ca64f489886008b02eba510bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8600dd95d7d441f9aaf8fc399cfccb14"
I20260929 18:37:50.496630 20754 raft_consensus.cc:3037] T 2c3d9d44a48742319edb4aebcbe8e71c P 8600dd95d7d441f9aaf8fc399cfccb14 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:37:50.496631 21027 raft_consensus.cc:3037] T 2c3d9d44a48742319edb4aebcbe8e71c P 2fbaf12215ba453bb80f9cccaaff2e73 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:37:50.497812 21027 raft_consensus.cc:2454] T 2c3d9d44a48742319edb4aebcbe8e71c P 2fbaf12215ba453bb80f9cccaaff2e73 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a579500ca64f489886008b02eba510bf in term 1.
I20260929 18:37:50.497812 20754 raft_consensus.cc:2454] T 2c3d9d44a48742319edb4aebcbe8e71c P 8600dd95d7d441f9aaf8fc399cfccb14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a579500ca64f489886008b02eba510bf in term 1.
I20260929 18:37:50.498109 20826 leader_election.cc:302] T 2c3d9d44a48742319edb4aebcbe8e71c P a579500ca64f489886008b02eba510bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8600dd95d7d441f9aaf8fc399cfccb14, a579500ca64f489886008b02eba510bf; no voters:
I20260929 18:37:50.498313 21114 raft_consensus.cc:2781] T 2c3d9d44a48742319edb4aebcbe8e71c P a579500ca64f489886008b02eba510bf [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:37:50.498518 21114 raft_consensus.cc:686] T 2c3d9d44a48742319edb4aebcbe8e71c P a579500ca64f489886008b02eba510bf [term 1 LEADER]: Becoming Leader. State: Replica: a579500ca64f489886008b02eba510bf, State: Running, Role: LEADER
I20260929 18:37:50.498868 21114 consensus_queue.cc:237] T 2c3d9d44a48742319edb4aebcbe8e71c P a579500ca64f489886008b02eba510bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fbaf12215ba453bb80f9cccaaff2e73" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 35631 } } peers { permanent_uuid: "8600dd95d7d441f9aaf8fc399cfccb14" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 38831 } } peers { permanent_uuid: "a579500ca64f489886008b02eba510bf" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 45533 } }
I20260929 18:37:50.501997 19836 catalog_manager.cc:5495] T 2c3d9d44a48742319edb4aebcbe8e71c P a579500ca64f489886008b02eba510bf reported cstate change: term changed from 0 to 1, leader changed from <none> to a579500ca64f489886008b02eba510bf (127.13.100.2). New cstate: current_term: 1 leader_uuid: "a579500ca64f489886008b02eba510bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fbaf12215ba453bb80f9cccaaff2e73" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 35631 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8600dd95d7d441f9aaf8fc399cfccb14" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 38831 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a579500ca64f489886008b02eba510bf" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 45533 } health_report { overall_health: HEALTHY } } }
I20260929 18:37:50.503671 21061 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16cac5b4dfe74684861c10d815fb051c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "16cac5b4dfe74684861c10d815fb051c" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "16cac5b4dfe74684861c10d815fb051c" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 34955 } } peers { permanent_uuid: "30e554990e1d47fb8e2b183647f95d5d" member_type: VOTER last_known_addr { host: "127.13.100.61" port: 46389 } attrs { promote: false } } peers { permanent_uuid: "cdfc2d79001d4a34ae6eddb06d8a0c0d" member_type: NON_VOTER last_known_addr { host: "127.13.100.60" port: 40697 } attrs { promote: true } } }
I20260929 18:37:50.503806 21061 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16cac5b4dfe74684861c10d815fb051c [sys.catalog]: This master's current role is: LEADER
I20260929 18:37:50.505303 21119 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 16cac5b4dfe74684861c10d815fb051c: attempting to promote NON_VOTER cdfc2d79001d4a34ae6eddb06d8a0c0d to VOTER
I20260929 18:37:50.505648 21119 consensus_queue.cc:237] T 00000000000000000000000000000000 P 16cac5b4dfe74684861c10d815fb051c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "16cac5b4dfe74684861c10d815fb051c" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 34955 } } peers { permanent_uuid: "30e554990e1d47fb8e2b183647f95d5d" member_type: VOTER last_known_addr { host: "127.13.100.61" port: 46389 } attrs { promote: false } } peers { permanent_uuid: "cdfc2d79001d4a34ae6eddb06d8a0c0d" member_type: VOTER last_known_addr { host: "127.13.100.60" port: 40697 } attrs { promote: false } }
I20260929 18:37:50.506742 20448 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 30e554990e1d47fb8e2b183647f95d5d [term 1 FOLLOWER]: Refusing update from remote peer 16cac5b4dfe74684861c10d815fb051c: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260929 18:37:50.507090 21118 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 16cac5b4dfe74684861c10d815fb051c [LEADER]: Connected to new peer: Peer: permanent_uuid: "30e554990e1d47fb8e2b183647f95d5d" member_type: VOTER last_known_addr { host: "127.13.100.61" port: 46389 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260929 18:37:50.507371 20631 raft_consensus.cc:1261] T 00000000000000000000000000000000 P cdfc2d79001d4a34ae6eddb06d8a0c0d [term 1 LEARNER]: Refusing update from remote peer 16cac5b4dfe74684861c10d815fb051c: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260929 18:37:50.507699 21118 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 16cac5b4dfe74684861c10d815fb051c [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdfc2d79001d4a34ae6eddb06d8a0c0d" member_type: VOTER last_known_addr { host: "127.13.100.60" port: 40697 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.000s
I20260929 18:37:50.508718 21092 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 16cac5b4dfe74684861c10d815fb051c [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 7 to 11, cdfc2d79001d4a34ae6eddb06d8a0c0d (127.13.100.60) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "16cac5b4dfe74684861c10d815fb051c" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 34955 } } peers { permanent_uuid: "30e554990e1d47fb8e2b183647f95d5d" member_type: VOTER last_known_addr { host: "127.13.100.61" port: 46389 } attrs { promote: false } } peers { permanent_uuid: "cdfc2d79001d4a34ae6eddb06d8a0c0d" member_type: VOTER last_known_addr { host: "127.13.100.60" port: 40697 } attrs { promote: false } } }
I20260929 18:37:50.509121 20448 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 30e554990e1d47fb8e2b183647f95d5d [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, cdfc2d79001d4a34ae6eddb06d8a0c0d (127.13.100.60) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "16cac5b4dfe74684861c10d815fb051c" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 34955 } } peers { permanent_uuid: "30e554990e1d47fb8e2b183647f95d5d" member_type: VOTER last_known_addr { host: "127.13.100.61" port: 46389 } attrs { promote: false } } peers { permanent_uuid: "cdfc2d79001d4a34ae6eddb06d8a0c0d" member_type: VOTER last_known_addr { host: "127.13.100.60" port: 40697 } attrs { promote: false } } }
I20260929 18:37:50.509874 20631 raft_consensus.cc:2932] T 00000000000000000000000000000000 P cdfc2d79001d4a34ae6eddb06d8a0c0d [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, cdfc2d79001d4a34ae6eddb06d8a0c0d (127.13.100.60) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "16cac5b4dfe74684861c10d815fb051c" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 34955 } } peers { permanent_uuid: "30e554990e1d47fb8e2b183647f95d5d" member_type: VOTER last_known_addr { host: "127.13.100.61" port: 46389 } attrs { promote: false } } peers { permanent_uuid: "cdfc2d79001d4a34ae6eddb06d8a0c0d" member_type: VOTER last_known_addr { host: "127.13.100.60" port: 40697 } attrs { promote: false } } }
I20260929 18:37:50.510277 21092 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16cac5b4dfe74684861c10d815fb051c [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "16cac5b4dfe74684861c10d815fb051c" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "16cac5b4dfe74684861c10d815fb051c" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 34955 } } peers { permanent_uuid: "30e554990e1d47fb8e2b183647f95d5d" member_type: VOTER last_known_addr { host: "127.13.100.61" port: 46389 } attrs { promote: false } } peers { permanent_uuid: "cdfc2d79001d4a34ae6eddb06d8a0c0d" member_type: VOTER last_known_addr { host: "127.13.100.60" port: 40697 } attrs { promote: false } } }
I20260929 18:37:50.510371 21092 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16cac5b4dfe74684861c10d815fb051c [sys.catalog]: This master's current role is: LEADER
I20260929 18:37:50.510533 21118 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16cac5b4dfe74684861c10d815fb051c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "16cac5b4dfe74684861c10d815fb051c" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "16cac5b4dfe74684861c10d815fb051c" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 34955 } } peers { permanent_uuid: "30e554990e1d47fb8e2b183647f95d5d" member_type: VOTER last_known_addr { host: "127.13.100.61" port: 46389 } attrs { promote: false } } peers { permanent_uuid: "cdfc2d79001d4a34ae6eddb06d8a0c0d" member_type: VOTER last_known_addr { host: "127.13.100.60" port: 40697 } attrs { promote: false } } }
I20260929 18:37:50.510659 21118 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16cac5b4dfe74684861c10d815fb051c [sys.catalog]: This master's current role is: LEADER
I20260929 18:37:50.510746 21092 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16cac5b4dfe74684861c10d815fb051c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "16cac5b4dfe74684861c10d815fb051c" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "16cac5b4dfe74684861c10d815fb051c" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 34955 } } peers { permanent_uuid: "30e554990e1d47fb8e2b183647f95d5d" member_type: VOTER last_known_addr { host: "127.13.100.61" port: 46389 } attrs { promote: false } } peers { permanent_uuid: "cdfc2d79001d4a34ae6eddb06d8a0c0d" member_type: VOTER last_known_addr { host: "127.13.100.60" port: 40697 } attrs { promote: false } } }
I20260929 18:37:50.510824 21092 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16cac5b4dfe74684861c10d815fb051c [sys.catalog]: This master's current role is: LEADER
I20260929 18:37:50.511106 21121 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30e554990e1d47fb8e2b183647f95d5d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "16cac5b4dfe74684861c10d815fb051c" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "16cac5b4dfe74684861c10d815fb051c" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 34955 } } peers { permanent_uuid: "30e554990e1d47fb8e2b183647f95d5d" member_type: VOTER last_known_addr { host: "127.13.100.61" port: 46389 } attrs { promote: false } } peers { permanent_uuid: "cdfc2d79001d4a34ae6eddb06d8a0c0d" member_type: VOTER last_known_addr { host: "127.13.100.60" port: 40697 } attrs { promote: false } } }
I20260929 18:37:50.511215 21121 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30e554990e1d47fb8e2b183647f95d5d [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:37:50.511569 21104 sys_catalog.cc:455] T 00000000000000000000000000000000 P cdfc2d79001d4a34ae6eddb06d8a0c0d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "16cac5b4dfe74684861c10d815fb051c" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "16cac5b4dfe74684861c10d815fb051c" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 34955 } } peers { permanent_uuid: "30e554990e1d47fb8e2b183647f95d5d" member_type: VOTER last_known_addr { host: "127.13.100.61" port: 46389 } attrs { promote: false } } peers { permanent_uuid: "cdfc2d79001d4a34ae6eddb06d8a0c0d" member_type: VOTER last_known_addr { host: "127.13.100.60" port: 40697 } attrs { promote: false } } }
I20260929 18:37:50.511654 21104 sys_catalog.cc:458] T 00000000000000000000000000000000 P cdfc2d79001d4a34ae6eddb06d8a0c0d [sys.catalog]: This master's current role is: FOLLOWER
W20260929 18:37:50.518775 21077 tablet.cc:2369] T 2c3d9d44a48742319edb4aebcbe8e71c P 2fbaf12215ba453bb80f9cccaaff2e73: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:37:50.554698 21101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:50.551432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39452 (local address 127.13.100.2:45533)
0929 18:37:50.551570 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:50.551574 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:50.551586 (+ 12us) server_negotiation.cc:407] Connection header received
0929 18:37:50.551672 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:50.551674 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:50.551725 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:50.551817 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:37:50.552032 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:50.552534 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:50.553346 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:50.553550 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:50.553647 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:50.553947 (+ 300us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:37:50.554349 (+ 402us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:37:50.554439 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:50.554529 (+ 90us) server_negotiation.cc:299] Negotiation successful
0929 18:37:50.554575 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 18:37:50.560374 21027 raft_consensus.cc:1261] T 2c3d9d44a48742319edb4aebcbe8e71c P 2fbaf12215ba453bb80f9cccaaff2e73 [term 1 FOLLOWER]: Refusing update from remote peer a579500ca64f489886008b02eba510bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:37:50.560374 20754 raft_consensus.cc:1261] T 2c3d9d44a48742319edb4aebcbe8e71c P 8600dd95d7d441f9aaf8fc399cfccb14 [term 1 FOLLOWER]: Refusing update from remote peer a579500ca64f489886008b02eba510bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:37:50.561101 21116 consensus_queue.cc:1035] T 2c3d9d44a48742319edb4aebcbe8e71c P a579500ca64f489886008b02eba510bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "2fbaf12215ba453bb80f9cccaaff2e73" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 35631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:37:50.561313 21114 consensus_queue.cc:1035] T 2c3d9d44a48742319edb4aebcbe8e71c P a579500ca64f489886008b02eba510bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "8600dd95d7d441f9aaf8fc399cfccb14" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 38831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:37:50.571156 21124 mvcc.cc:205] Tried to move back new op lower bound from 7334736161007611904 to 7334736160765681664. Current Snapshot: MvccSnapshot[applied={T|T < 7334736161007611904}]
I20260929 18:37:50.574200 21123 mvcc.cc:205] Tried to move back new op lower bound from 7334736161007611904 to 7334736160765681664. Current Snapshot: MvccSnapshot[applied={T|T < 7334736161007611904 or (T in {7334736161007611904})}]
I20260929 18:37:50.578704 19836 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51558:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:37:50.578881 19836 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51558:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:37:50.583483 19836 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 16cac5b4dfe74684861c10d815fb051c: Sending DeleteTablet for 3 replicas of tablet 2c3d9d44a48742319edb4aebcbe8e71c
I20260929 18:37:50.584905 20871 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c3d9d44a48742319edb4aebcbe8e71c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:37:50 UTC) from {username='oryx', principal='oryx/127.13.100.62@KRBTEST.COM'} at 127.0.0.1:39440
I20260929 18:37:50.584928 21007 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c3d9d44a48742319edb4aebcbe8e71c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:37:50 UTC) from {username='oryx', principal='oryx/127.13.100.62@KRBTEST.COM'} at 127.0.0.1:55552
I20260929 18:37:50.585031 20734 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c3d9d44a48742319edb4aebcbe8e71c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:37:50 UTC) from {username='oryx', principal='oryx/127.13.100.62@KRBTEST.COM'} at 127.0.0.1:42818
I20260929 18:37:50.585453 21133 tablet_replica.cc:330] T 2c3d9d44a48742319edb4aebcbe8e71c P a579500ca64f489886008b02eba510bf: stopping tablet replica
I20260929 18:37:50.585759 21133 raft_consensus.cc:2229] T 2c3d9d44a48742319edb4aebcbe8e71c P a579500ca64f489886008b02eba510bf [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:37:50.586156 21133 raft_consensus.cc:2258] T 2c3d9d44a48742319edb4aebcbe8e71c P a579500ca64f489886008b02eba510bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:37:50.586484 21134 tablet_replica.cc:330] T 2c3d9d44a48742319edb4aebcbe8e71c P 2fbaf12215ba453bb80f9cccaaff2e73: stopping tablet replica
I20260929 18:37:50.586798 21134 raft_consensus.cc:2229] T 2c3d9d44a48742319edb4aebcbe8e71c P 2fbaf12215ba453bb80f9cccaaff2e73 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:37:50.587122 21134 raft_consensus.cc:2258] T 2c3d9d44a48742319edb4aebcbe8e71c P 2fbaf12215ba453bb80f9cccaaff2e73 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:37:50.587980 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 19805
I20260929 18:37:50.588727 21133 ts_tablet_manager.cc:1905] T 2c3d9d44a48742319edb4aebcbe8e71c P a579500ca64f489886008b02eba510bf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:37:50.589556 21134 ts_tablet_manager.cc:1905] T 2c3d9d44a48742319edb4aebcbe8e71c P 2fbaf12215ba453bb80f9cccaaff2e73: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:37:50.591536 21133 ts_tablet_manager.cc:1918] T 2c3d9d44a48742319edb4aebcbe8e71c P a579500ca64f489886008b02eba510bf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:37:50.591640 21133 log.cc:1196] T 2c3d9d44a48742319edb4aebcbe8e71c P a579500ca64f489886008b02eba510bf: Deleting WAL directory at /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/ts-1/wal/wals/2c3d9d44a48742319edb4aebcbe8e71c
I20260929 18:37:50.591970 21133 ts_tablet_manager.cc:1939] T 2c3d9d44a48742319edb4aebcbe8e71c P a579500ca64f489886008b02eba510bf: Deleting consensus metadata
I20260929 18:37:50.592212 21134 ts_tablet_manager.cc:1918] T 2c3d9d44a48742319edb4aebcbe8e71c P 2fbaf12215ba453bb80f9cccaaff2e73: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:37:50.592303 21134 log.cc:1196] T 2c3d9d44a48742319edb4aebcbe8e71c P 2fbaf12215ba453bb80f9cccaaff2e73: Deleting WAL directory at /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/ts-2/wal/wals/2c3d9d44a48742319edb4aebcbe8e71c
I20260929 18:37:50.592603 21134 ts_tablet_manager.cc:1939] T 2c3d9d44a48742319edb4aebcbe8e71c P 2fbaf12215ba453bb80f9cccaaff2e73: Deleting consensus metadata
I20260929 18:37:50.593318 21135 tablet_replica.cc:330] T 2c3d9d44a48742319edb4aebcbe8e71c P 8600dd95d7d441f9aaf8fc399cfccb14: stopping tablet replica
I20260929 18:37:50.593667 21135 raft_consensus.cc:2229] T 2c3d9d44a48742319edb4aebcbe8e71c P 8600dd95d7d441f9aaf8fc399cfccb14 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:37:50.594020 21135 raft_consensus.cc:2258] T 2c3d9d44a48742319edb4aebcbe8e71c P 8600dd95d7d441f9aaf8fc399cfccb14 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260929 18:37:50.596443 20826 connection.cc:535] server connection from 127.0.0.1:39440 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260929 18:37:50.596786 20960 connection.cc:535] server connection from 127.0.0.1:55552 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 18:37:50.603087 21088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:50.586914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48344 (local address 127.13.100.61:46389)
0929 18:37:50.587027 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:50.587032 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:50.593339 (+ 6307us) server_negotiation.cc:407] Connection header received
0929 18:37:50.593542 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:50.593545 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:50.593622 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:50.593678 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:50.594728 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:50.595264 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:50.596112 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:50.596349 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:50.596482 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:50.599000 (+ 2518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:50.599003 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:50.599006 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:37:50.599038 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:50.601193 (+ 2155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:50.601677 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:50.601682 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:50.601684 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:50.601775 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:50.602130 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:50.602133 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:50.602136 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:50.602341 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:50.602466 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:50.602815 (+ 349us) server_negotiation.cc:299] Negotiation successful
0929 18:37:50.602931 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260929 18:37:50.603188 21135 ts_tablet_manager.cc:1905] T 2c3d9d44a48742319edb4aebcbe8e71c P 8600dd95d7d441f9aaf8fc399cfccb14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:37:50.606436 21135 ts_tablet_manager.cc:1918] T 2c3d9d44a48742319edb4aebcbe8e71c P 8600dd95d7d441f9aaf8fc399cfccb14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:37:50.606539 21135 log.cc:1196] T 2c3d9d44a48742319edb4aebcbe8e71c P 8600dd95d7d441f9aaf8fc399cfccb14: Deleting WAL directory at /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/ts-0/wal/wals/2c3d9d44a48742319edb4aebcbe8e71c
I20260929 18:37:50.606876 21135 ts_tablet_manager.cc:1939] T 2c3d9d44a48742319edb4aebcbe8e71c P 8600dd95d7d441f9aaf8fc399cfccb14: Deleting consensus metadata
W20260929 18:37:50.608151 20687 connection.cc:441] server connection from 127.0.0.1:42818 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42818 (request call id 1) could send its response
I20260929 18:37:50.608844 20560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:50.599898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48346 (local address 127.13.100.61:46389)
0929 18:37:50.600012 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:50.600017 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:50.600029 (+ 12us) server_negotiation.cc:407] Connection header received
0929 18:37:50.600060 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:50.600063 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:50.600101 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:50.600168 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:50.600948 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:50.601729 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:50.602826 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:50.603117 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:50.603187 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:50.604927 (+ 1740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:50.604931 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:50.604935 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:37:50.604962 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:50.606780 (+ 1818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:50.607575 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:50.607578 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:50.607580 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:50.607661 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:50.607953 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:50.607956 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:50.607958 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:50.608156 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:50.608257 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:50.608586 (+ 329us) server_negotiation.cc:299] Negotiation successful
0929 18:37:50.608691 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 18:37:50.612519 20804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:50.587155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54498 (local address 127.13.100.60:40697)
0929 18:37:50.587260 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:50.587265 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:50.603541 (+ 16276us) server_negotiation.cc:407] Connection header received
0929 18:37:50.603998 (+ 457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:50.604001 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:50.604047 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:50.604094 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:50.604888 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:50.605728 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:50.606889 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:50.607227 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:50.607302 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:50.608962 (+ 1660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:50.608970 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:50.608976 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:37:50.609004 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:50.610850 (+ 1846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:50.611296 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:50.611302 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:50.611304 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:50.611385 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:50.611670 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:50.611677 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:50.611680 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:50.611892 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:50.612019 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:50.612244 (+ 225us) server_negotiation.cc:299] Negotiation successful
0929 18:37:50.612361 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 18:37:50.613380 21089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:50.600558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54512 (local address 127.13.100.60:40697)
0929 18:37:50.600692 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:50.600704 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:50.601112 (+ 408us) server_negotiation.cc:407] Connection header received
0929 18:37:50.601300 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:50.601303 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:50.601360 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:50.601426 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:50.602778 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:50.603530 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:50.606039 (+ 2509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:50.606407 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:50.606484 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:50.608566 (+ 2082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:50.608571 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:50.608575 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:37:50.608603 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:50.611389 (+ 2786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:50.611852 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:50.611855 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:50.611857 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:50.611929 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:50.612245 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:50.612249 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:50.612252 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:50.612431 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:50.612519 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:50.613176 (+ 657us) server_negotiation.cc:299] Negotiation successful
0929 18:37:50.613283 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 18:37:51.573355 21143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:51.572681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:34955 (local address 127.13.100.1:35411)
0929 18:37:51.573130 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:51.573237 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.62:34955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":78,"threads_started":1}
W20260929 18:37:51.573730 20801 heartbeater.cc:644] Failed to heartbeat to 127.13.100.62:34955 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.13.100.62:34955: connect: Connection refused (error 111)
I20260929 18:37:51.574962 21144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:51.574526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:34955 (local address 127.13.100.3:45469)
0929 18:37:51.574758 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:51.574853 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.62:34955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":78,"threads_started":1}
W20260929 18:37:51.576303 21075 heartbeater.cc:644] Failed to heartbeat to 127.13.100.62:34955 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.13.100.62:34955: connect: Connection refused (error 111)
I20260929 18:37:51.576510 21145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:51.576008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:34955 (local address 127.13.100.2:44045)
0929 18:37:51.576333 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:51.576415 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.62:34955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":59,"threads_started":1}
W20260929 18:37:51.576766 20938 heartbeater.cc:644] Failed to heartbeat to 127.13.100.62:34955 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.13.100.62:34955: connect: Connection refused (error 111)
I20260929 18:37:52.084288 21146 raft_consensus.cc:482] T 00000000000000000000000000000000 P 30e554990e1d47fb8e2b183647f95d5d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 16cac5b4dfe74684861c10d815fb051c)
I20260929 18:37:52.084466 21147 raft_consensus.cc:482] T 00000000000000000000000000000000 P cdfc2d79001d4a34ae6eddb06d8a0c0d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 16cac5b4dfe74684861c10d815fb051c)
I20260929 18:37:52.084529 21146 raft_consensus.cc:504] T 00000000000000000000000000000000 P 30e554990e1d47fb8e2b183647f95d5d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "16cac5b4dfe74684861c10d815fb051c" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 34955 } } peers { permanent_uuid: "30e554990e1d47fb8e2b183647f95d5d" member_type: VOTER last_known_addr { host: "127.13.100.61" port: 46389 } attrs { promote: false } } peers { permanent_uuid: "cdfc2d79001d4a34ae6eddb06d8a0c0d" member_type: VOTER last_known_addr { host: "127.13.100.60" port: 40697 } attrs { promote: false } }
I20260929 18:37:52.084707 21147 raft_consensus.cc:504] T 00000000000000000000000000000000 P cdfc2d79001d4a34ae6eddb06d8a0c0d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "16cac5b4dfe74684861c10d815fb051c" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 34955 } } peers { permanent_uuid: "30e554990e1d47fb8e2b183647f95d5d" member_type: VOTER last_known_addr { host: "127.13.100.61" port: 46389 } attrs { promote: false } } peers { permanent_uuid: "cdfc2d79001d4a34ae6eddb06d8a0c0d" member_type: VOTER last_known_addr { host: "127.13.100.60" port: 40697 } attrs { promote: false } }
I20260929 18:37:52.085628 21146 leader_election.cc:288] T 00000000000000000000000000000000 P 30e554990e1d47fb8e2b183647f95d5d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 16cac5b4dfe74684861c10d815fb051c (127.13.100.62:34955), cdfc2d79001d4a34ae6eddb06d8a0c0d (127.13.100.60:40697)
I20260929 18:37:52.085928 21147 leader_election.cc:288] T 00000000000000000000000000000000 P cdfc2d79001d4a34ae6eddb06d8a0c0d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 16cac5b4dfe74684861c10d815fb051c (127.13.100.62:34955), 30e554990e1d47fb8e2b183647f95d5d (127.13.100.61:46389)
I20260929 18:37:52.086419 21148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:52.085767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:34955 (local address 127.0.0.1:39448)
0929 18:37:52.086222 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:52.086318 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.62:34955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":59,"threads_started":1}
W20260929 18:37:52.086601 20607 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.100.62:34955: connect: Connection refused (error 111)
I20260929 18:37:52.086948 21152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:52.085766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:34955 (local address 127.0.0.1:39446)
0929 18:37:52.086333 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:52.086391 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.62:34955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":79,"threads_started":1}
W20260929 18:37:52.087101 20424 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.100.62:34955: connect: Connection refused (error 111)
I20260929 18:37:52.087569 21148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:52.087313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:34955 (local address 127.0.0.1:39460)
0929 18:37:52.087452 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:52.087503 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.62:34955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
W20260929 18:37:52.087672 20607 leader_election.cc:334] T 00000000000000000000000000000000 P cdfc2d79001d4a34ae6eddb06d8a0c0d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 16cac5b4dfe74684861c10d815fb051c (127.13.100.62:34955): Network error: Client connection negotiation failed: client connection to 127.13.100.62:34955: connect: Connection refused (error 111)
I20260929 18:37:52.087980 21152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:52.087749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:34955 (local address 127.0.0.1:39474)
0929 18:37:52.087880 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:52.087928 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.62:34955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
W20260929 18:37:52.088131 20424 leader_election.cc:334] T 00000000000000000000000000000000 P 30e554990e1d47fb8e2b183647f95d5d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 16cac5b4dfe74684861c10d815fb051c (127.13.100.62:34955): Network error: Client connection negotiation failed: client connection to 127.13.100.62:34955: connect: Connection refused (error 111)
I20260929 18:37:52.091171 21150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:52.085986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.61:46389 (local address 127.0.0.1:56074)
0929 18:37:52.086259 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:52.086274 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:52.086378 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:52.086866 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:52.086869 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:52.086900 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:37:52.087272 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:52.087279 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:52.088378 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:52.088381 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:52.089680 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:52.089691 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:52.090745 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:52.090749 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:52.090845 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:52.090870 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 18:37:52.090962 (+ 92us) client_negotiation.cc:240] Negotiation successful
0929 18:37:52.091034 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"mutex_wait_us":200,"spinlock_wait_cycles":6400,"thread_start_us":63,"threads_started":1}
I20260929 18:37:52.091171 21149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:52.085766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.60:40697 (local address 127.0.0.1:33800)
0929 18:37:52.086407 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:52.086422 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:52.086604 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:52.086962 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:52.086965 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:52.086981 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:37:52.087476 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:52.087482 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:52.088535 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:52.088539 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:52.089729 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:52.089741 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:52.090758 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:52.090762 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:52.090848 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:52.090870 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 18:37:52.090962 (+ 92us) client_negotiation.cc:240] Negotiation successful
0929 18:37:52.091034 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":580,"mutex_wait_us":271,"thread_start_us":61,"threads_started":1}
I20260929 18:37:52.091313 21151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:52.085998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33800 (local address 127.13.100.60:40697)
0929 18:37:52.086196 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:52.086200 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:52.086485 (+ 285us) server_negotiation.cc:407] Connection header received
0929 18:37:52.086733 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:52.086736 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:52.086783 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:52.086879 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:37:52.087592 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:52.088433 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:52.089846 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:52.090597 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:52.090713 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:52.090772 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:52.091024 (+ 252us) server_negotiation.cc:299] Negotiation successful
0929 18:37:52.091105 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":120,"spinlock_wait_cycles":2688,"thread_start_us":57,"threads_started":1}
I20260929 18:37:52.091274 21153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:52.086030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56074 (local address 127.13.100.61:46389)
0929 18:37:52.086617 (+ 587us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:52.086621 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:52.086633 (+ 12us) server_negotiation.cc:407] Connection header received
0929 18:37:52.086678 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:52.086680 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:52.086736 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:52.086844 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:37:52.087386 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:52.088269 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:52.089846 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:52.090600 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:52.090713 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:52.090772 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:52.091025 (+ 253us) server_negotiation.cc:299] Negotiation successful
0929 18:37:52.091107 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":519,"thread_start_us":62,"threads_started":1}
I20260929 18:37:52.092023 20448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cdfc2d79001d4a34ae6eddb06d8a0c0d" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "30e554990e1d47fb8e2b183647f95d5d" is_pre_election: true
I20260929 18:37:52.092047 20631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "30e554990e1d47fb8e2b183647f95d5d" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "cdfc2d79001d4a34ae6eddb06d8a0c0d" is_pre_election: true
I20260929 18:37:52.092243 20448 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 30e554990e1d47fb8e2b183647f95d5d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cdfc2d79001d4a34ae6eddb06d8a0c0d in term 1.
I20260929 18:37:52.092238 20631 raft_consensus.cc:2454] T 00000000000000000000000000000000 P cdfc2d79001d4a34ae6eddb06d8a0c0d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30e554990e1d47fb8e2b183647f95d5d in term 1.
I20260929 18:37:52.092587 20605 leader_election.cc:302] T 00000000000000000000000000000000 P cdfc2d79001d4a34ae6eddb06d8a0c0d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30e554990e1d47fb8e2b183647f95d5d, cdfc2d79001d4a34ae6eddb06d8a0c0d; no voters: 16cac5b4dfe74684861c10d815fb051c
I20260929 18:37:52.092608 20425 leader_election.cc:302] T 00000000000000000000000000000000 P 30e554990e1d47fb8e2b183647f95d5d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30e554990e1d47fb8e2b183647f95d5d, cdfc2d79001d4a34ae6eddb06d8a0c0d; no voters: 16cac5b4dfe74684861c10d815fb051c
I20260929 18:37:52.092803 21147 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cdfc2d79001d4a34ae6eddb06d8a0c0d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260929 18:37:52.092861 21147 raft_consensus.cc:482] T 00000000000000000000000000000000 P cdfc2d79001d4a34ae6eddb06d8a0c0d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 16cac5b4dfe74684861c10d815fb051c)
I20260929 18:37:52.092900 21147 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cdfc2d79001d4a34ae6eddb06d8a0c0d [term 1 FOLLOWER]: Advancing to term 2
I20260929 18:37:52.093057 21146 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 30e554990e1d47fb8e2b183647f95d5d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260929 18:37:52.093150 21146 raft_consensus.cc:482] T 00000000000000000000000000000000 P 30e554990e1d47fb8e2b183647f95d5d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 16cac5b4dfe74684861c10d815fb051c)
I20260929 18:37:52.093212 21146 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 30e554990e1d47fb8e2b183647f95d5d [term 1 FOLLOWER]: Advancing to term 2
I20260929 18:37:52.094034 21147 raft_consensus.cc:504] T 00000000000000000000000000000000 P cdfc2d79001d4a34ae6eddb06d8a0c0d [term 2 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "16cac5b4dfe74684861c10d815fb051c" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 34955 } } peers { permanent_uuid: "30e554990e1d47fb8e2b183647f95d5d" member_type: VOTER last_known_addr { host: "127.13.100.61" port: 46389 } attrs { promote: false } } peers { permanent_uuid: "cdfc2d79001d4a34ae6eddb06d8a0c0d" member_type: VOTER last_known_addr { host: "127.13.100.60" port: 40697 } attrs { promote: false } }
I20260929 18:37:52.094259 21146 raft_consensus.cc:504] T 00000000000000000000000000000000 P 30e554990e1d47fb8e2b183647f95d5d [term 2 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "16cac5b4dfe74684861c10d815fb051c" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 34955 } } peers { permanent_uuid: "30e554990e1d47fb8e2b183647f95d5d" member_type: VOTER last_known_addr { host: "127.13.100.61" port: 46389 } attrs { promote: false } } peers { permanent_uuid: "cdfc2d79001d4a34ae6eddb06d8a0c0d" member_type: VOTER last_known_addr { host: "127.13.100.60" port: 40697 } attrs { promote: false } }
I20260929 18:37:52.094453 21147 leader_election.cc:288] T 00000000000000000000000000000000 P cdfc2d79001d4a34ae6eddb06d8a0c0d [CANDIDATE]: Term 2 election: Requested vote from peers 16cac5b4dfe74684861c10d815fb051c (127.13.100.62:34955), 30e554990e1d47fb8e2b183647f95d5d (127.13.100.61:46389)
I20260929 18:37:52.094739 21146 leader_election.cc:288] T 00000000000000000000000000000000 P 30e554990e1d47fb8e2b183647f95d5d [CANDIDATE]: Term 2 election: Requested vote from peers 16cac5b4dfe74684861c10d815fb051c (127.13.100.62:34955), cdfc2d79001d4a34ae6eddb06d8a0c0d (127.13.100.60:40697)
I20260929 18:37:52.094775 21150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:52.094540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:34955 (local address 127.0.0.1:39486)
0929 18:37:52.094623 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:52.094674 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.62:34955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260929 18:37:52.094914 20631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "30e554990e1d47fb8e2b183647f95d5d" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "cdfc2d79001d4a34ae6eddb06d8a0c0d"
I20260929 18:37:52.094964 20448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cdfc2d79001d4a34ae6eddb06d8a0c0d" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "30e554990e1d47fb8e2b183647f95d5d"
I20260929 18:37:52.095094 20631 raft_consensus.cc:2379] T 00000000000000000000000000000000 P cdfc2d79001d4a34ae6eddb06d8a0c0d [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 30e554990e1d47fb8e2b183647f95d5d in current term 2: Already voted for candidate cdfc2d79001d4a34ae6eddb06d8a0c0d in this term.
I20260929 18:37:52.095139 20448 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 30e554990e1d47fb8e2b183647f95d5d [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate cdfc2d79001d4a34ae6eddb06d8a0c0d in current term 2: Already voted for candidate 30e554990e1d47fb8e2b183647f95d5d in this term.
I20260929 18:37:52.095408 21149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:52.094720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:34955 (local address 127.0.0.1:39488)
0929 18:37:52.095286 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:52.095340 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.62:34955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260929 18:37:52.095726 21150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:52.095207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:34955 (local address 127.0.0.1:39500)
0929 18:37:52.095634 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:52.095679 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.62:34955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260929 18:37:52.095829 20607 leader_election.cc:334] T 00000000000000000000000000000000 P cdfc2d79001d4a34ae6eddb06d8a0c0d [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 16cac5b4dfe74684861c10d815fb051c (127.13.100.62:34955): Network error: Client connection negotiation failed: client connection to 127.13.100.62:34955: connect: Connection refused (error 111)
I20260929 18:37:52.095890 20607 leader_election.cc:302] T 00000000000000000000000000000000 P cdfc2d79001d4a34ae6eddb06d8a0c0d [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cdfc2d79001d4a34ae6eddb06d8a0c0d; no voters: 16cac5b4dfe74684861c10d815fb051c, 30e554990e1d47fb8e2b183647f95d5d
I20260929 18:37:52.096032 21147 raft_consensus.cc:2726] T 00000000000000000000000000000000 P cdfc2d79001d4a34ae6eddb06d8a0c0d [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260929 18:37:52.096191 21149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:52.095852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:34955 (local address 127.0.0.1:39516)
0929 18:37:52.096010 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:52.096060 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.62:34955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260929 18:37:52.096303 20424 leader_election.cc:334] T 00000000000000000000000000000000 P 30e554990e1d47fb8e2b183647f95d5d [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 16cac5b4dfe74684861c10d815fb051c (127.13.100.62:34955): Network error: Client connection negotiation failed: client connection to 127.13.100.62:34955: connect: Connection refused (error 111)
I20260929 18:37:52.096374 20424 leader_election.cc:302] T 00000000000000000000000000000000 P 30e554990e1d47fb8e2b183647f95d5d [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30e554990e1d47fb8e2b183647f95d5d; no voters: 16cac5b4dfe74684861c10d815fb051c, cdfc2d79001d4a34ae6eddb06d8a0c0d
I20260929 18:37:52.096519 21146 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 30e554990e1d47fb8e2b183647f95d5d [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260929 18:37:52.574856 21156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:52.574336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:34955 (local address 127.13.100.1:49721)
0929 18:37:52.574697 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:52.574762 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.62:34955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":78,"threads_started":1}
I20260929 18:37:52.577266 21157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:52.576828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:34955 (local address 127.13.100.3:35369)
0929 18:37:52.577108 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:52.577171 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.62:34955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":97,"threads_started":1}
I20260929 18:37:52.577713 21158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:52.577272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:34955 (local address 127.13.100.2:49123)
0929 18:37:52.577573 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:52.577639 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.62:34955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":66,"threads_started":1}
I20260929 18:37:53.576017 21159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:53.575554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:34955 (local address 127.13.100.1:38271)
0929 18:37:53.575855 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:53.575929 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.62:34955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":77,"threads_started":1}
I20260929 18:37:53.578482 21160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:53.578025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:34955 (local address 127.13.100.3:48731)
0929 18:37:53.578332 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:53.578384 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.62:34955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":53,"threads_started":1}
I20260929 18:37:53.578799 21161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:53.578371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:34955 (local address 127.13.100.2:34041)
0929 18:37:53.578651 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:53.578710 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.62:34955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":67,"threads_started":1}
I20260929 18:37:53.737327 21162 raft_consensus.cc:482] T 00000000000000000000000000000000 P cdfc2d79001d4a34ae6eddb06d8a0c0d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:37:53.737473 21162 raft_consensus.cc:504] T 00000000000000000000000000000000 P cdfc2d79001d4a34ae6eddb06d8a0c0d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "16cac5b4dfe74684861c10d815fb051c" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 34955 } } peers { permanent_uuid: "30e554990e1d47fb8e2b183647f95d5d" member_type: VOTER last_known_addr { host: "127.13.100.61" port: 46389 } attrs { promote: false } } peers { permanent_uuid: "cdfc2d79001d4a34ae6eddb06d8a0c0d" member_type: VOTER last_known_addr { host: "127.13.100.60" port: 40697 } attrs { promote: false } }
I20260929 18:37:53.737936 21162 leader_election.cc:288] T 00000000000000000000000000000000 P cdfc2d79001d4a34ae6eddb06d8a0c0d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 16cac5b4dfe74684861c10d815fb051c (127.13.100.62:34955), 30e554990e1d47fb8e2b183647f95d5d (127.13.100.61:46389)
I20260929 18:37:53.738257 20448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cdfc2d79001d4a34ae6eddb06d8a0c0d" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "30e554990e1d47fb8e2b183647f95d5d" is_pre_election: true
I20260929 18:37:53.738408 20448 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 30e554990e1d47fb8e2b183647f95d5d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cdfc2d79001d4a34ae6eddb06d8a0c0d in term 2.
I20260929 18:37:53.738498 21163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:53.738101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:34955 (local address 127.0.0.1:39728)
0929 18:37:53.738318 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:53.738367 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.62:34955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":56,"threads_started":1}
I20260929 18:37:53.738751 20605 leader_election.cc:302] T 00000000000000000000000000000000 P cdfc2d79001d4a34ae6eddb06d8a0c0d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30e554990e1d47fb8e2b183647f95d5d, cdfc2d79001d4a34ae6eddb06d8a0c0d; no voters:
I20260929 18:37:53.738917 21162 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cdfc2d79001d4a34ae6eddb06d8a0c0d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260929 18:37:53.738972 21162 raft_consensus.cc:482] T 00000000000000000000000000000000 P cdfc2d79001d4a34ae6eddb06d8a0c0d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:37:53.739025 21162 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cdfc2d79001d4a34ae6eddb06d8a0c0d [term 2 FOLLOWER]: Advancing to term 3
I20260929 18:37:53.739245 21163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:53.739049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:34955 (local address 127.0.0.1:39734)
0929 18:37:53.739150 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:53.739199 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.62:34955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260929 18:37:53.739341 20607 leader_election.cc:334] T 00000000000000000000000000000000 P cdfc2d79001d4a34ae6eddb06d8a0c0d [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 16cac5b4dfe74684861c10d815fb051c (127.13.100.62:34955): Network error: Client connection negotiation failed: client connection to 127.13.100.62:34955: connect: Connection refused (error 111)
I20260929 18:37:53.740077 21162 raft_consensus.cc:504] T 00000000000000000000000000000000 P cdfc2d79001d4a34ae6eddb06d8a0c0d [term 3 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "16cac5b4dfe74684861c10d815fb051c" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 34955 } } peers { permanent_uuid: "30e554990e1d47fb8e2b183647f95d5d" member_type: VOTER last_known_addr { host: "127.13.100.61" port: 46389 } attrs { promote: false } } peers { permanent_uuid: "cdfc2d79001d4a34ae6eddb06d8a0c0d" member_type: VOTER last_known_addr { host: "127.13.100.60" port: 40697 } attrs { promote: false } }
I20260929 18:37:53.740406 21162 leader_election.cc:288] T 00000000000000000000000000000000 P cdfc2d79001d4a34ae6eddb06d8a0c0d [CANDIDATE]: Term 3 election: Requested vote from peers 16cac5b4dfe74684861c10d815fb051c (127.13.100.62:34955), 30e554990e1d47fb8e2b183647f95d5d (127.13.100.61:46389)
I20260929 18:37:53.740682 21163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:53.740493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:34955 (local address 127.0.0.1:39744)
0929 18:37:53.740582 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:53.740627 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.62:34955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260929 18:37:53.740716 20448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cdfc2d79001d4a34ae6eddb06d8a0c0d" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "30e554990e1d47fb8e2b183647f95d5d"
I20260929 18:37:53.740830 20448 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 30e554990e1d47fb8e2b183647f95d5d [term 2 FOLLOWER]: Advancing to term 3
I20260929 18:37:53.741415 21163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:53.741187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:34955 (local address 127.0.0.1:39760)
0929 18:37:53.741304 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:53.741355 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.62:34955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260929 18:37:53.741526 20607 leader_election.cc:334] T 00000000000000000000000000000000 P cdfc2d79001d4a34ae6eddb06d8a0c0d [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 16cac5b4dfe74684861c10d815fb051c (127.13.100.62:34955): Network error: Client connection negotiation failed: client connection to 127.13.100.62:34955: connect: Connection refused (error 111)
I20260929 18:37:53.742031 20448 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 30e554990e1d47fb8e2b183647f95d5d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cdfc2d79001d4a34ae6eddb06d8a0c0d in term 3.
I20260929 18:37:53.742311 20605 leader_election.cc:302] T 00000000000000000000000000000000 P cdfc2d79001d4a34ae6eddb06d8a0c0d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30e554990e1d47fb8e2b183647f95d5d, cdfc2d79001d4a34ae6eddb06d8a0c0d; no voters: 16cac5b4dfe74684861c10d815fb051c
I20260929 18:37:53.742494 21162 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cdfc2d79001d4a34ae6eddb06d8a0c0d [term 3 FOLLOWER]: Leader election won for term 3
I20260929 18:37:53.742673 21162 raft_consensus.cc:686] T 00000000000000000000000000000000 P cdfc2d79001d4a34ae6eddb06d8a0c0d [term 3 LEADER]: Becoming Leader. State: Replica: cdfc2d79001d4a34ae6eddb06d8a0c0d, State: Running, Role: LEADER
I20260929 18:37:53.742980 21162 consensus_queue.cc:237] T 00000000000000000000000000000000 P cdfc2d79001d4a34ae6eddb06d8a0c0d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "16cac5b4dfe74684861c10d815fb051c" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 34955 } } peers { permanent_uuid: "30e554990e1d47fb8e2b183647f95d5d" member_type: VOTER last_known_addr { host: "127.13.100.61" port: 46389 } attrs { promote: false } } peers { permanent_uuid: "cdfc2d79001d4a34ae6eddb06d8a0c0d" member_type: VOTER last_known_addr { host: "127.13.100.60" port: 40697 } attrs { promote: false } }
I20260929 18:37:53.744009 21165 sys_catalog.cc:455] T 00000000000000000000000000000000 P cdfc2d79001d4a34ae6eddb06d8a0c0d [sys.catalog]: SysCatalogTable state changed. Reason: New leader cdfc2d79001d4a34ae6eddb06d8a0c0d. Latest consensus state: current_term: 3 leader_uuid: "cdfc2d79001d4a34ae6eddb06d8a0c0d" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "16cac5b4dfe74684861c10d815fb051c" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 34955 } } peers { permanent_uuid: "30e554990e1d47fb8e2b183647f95d5d" member_type: VOTER last_known_addr { host: "127.13.100.61" port: 46389 } attrs { promote: false } } peers { permanent_uuid: "cdfc2d79001d4a34ae6eddb06d8a0c0d" member_type: VOTER last_known_addr { host: "127.13.100.60" port: 40697 } attrs { promote: false } } }
I20260929 18:37:53.744145 21165 sys_catalog.cc:458] T 00000000000000000000000000000000 P cdfc2d79001d4a34ae6eddb06d8a0c0d [sys.catalog]: This master's current role is: LEADER
I20260929 18:37:53.744408 21167 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:37:53.746797 21167 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:37:53.747190 21167 catalog_manager.cc:1259] Loaded cluster ID: 3dd0e53705814f59aa70862dfbe1aea6
I20260929 18:37:53.747234 21167 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:37:53.747752 21167 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:37:53.748087 21167 catalog_manager.cc:5879] T 00000000000000000000000000000000 P cdfc2d79001d4a34ae6eddb06d8a0c0d: Loaded TSK: 0
I20260929 18:37:53.748826 21167 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:37:53.821547 20621 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54512:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:37:53.824786 20621 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:37:53.827668 21163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:53.827395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:34955 (local address 127.0.0.1:39790)
0929 18:37:53.827530 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:53.827588 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.62:34955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260929 18:37:53.827668 20448 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 30e554990e1d47fb8e2b183647f95d5d [term 3 FOLLOWER]: Refusing update from remote peer cdfc2d79001d4a34ae6eddb06d8a0c0d: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260929 18:37:53.828217 21165 consensus_queue.cc:1035] T 00000000000000000000000000000000 P cdfc2d79001d4a34ae6eddb06d8a0c0d [LEADER]: Connected to new peer: Peer: permanent_uuid: "30e554990e1d47fb8e2b183647f95d5d" member_type: VOTER last_known_addr { host: "127.13.100.61" port: 46389 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260929 18:37:53.828385 21163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:53.828158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:34955 (local address 127.0.0.1:39806)
0929 18:37:53.828268 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:53.828321 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.62:34955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260929 18:37:53.828487 20607 consensus_peers.cc:487] T 00000000000000000000000000000000 P cdfc2d79001d4a34ae6eddb06d8a0c0d -> Peer 16cac5b4dfe74684861c10d815fb051c (127.13.100.62:34955): Couldn't send request to peer 16cac5b4dfe74684861c10d815fb051c. Status: Network error: Client connection negotiation failed: client connection to 127.13.100.62:34955: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260929 18:37:53.829741 21169 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30e554990e1d47fb8e2b183647f95d5d [sys.catalog]: SysCatalogTable state changed. Reason: New leader cdfc2d79001d4a34ae6eddb06d8a0c0d. Latest consensus state: current_term: 3 leader_uuid: "cdfc2d79001d4a34ae6eddb06d8a0c0d" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "16cac5b4dfe74684861c10d815fb051c" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 34955 } } peers { permanent_uuid: "30e554990e1d47fb8e2b183647f95d5d" member_type: VOTER last_known_addr { host: "127.13.100.61" port: 46389 } attrs { promote: false } } peers { permanent_uuid: "cdfc2d79001d4a34ae6eddb06d8a0c0d" member_type: VOTER last_known_addr { host: "127.13.100.60" port: 40697 } attrs { promote: false } } }
I20260929 18:37:53.829862 21169 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30e554990e1d47fb8e2b183647f95d5d [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:37:53.830665 21168 mvcc.cc:205] Tried to move back new op lower bound from 7334736174393544704 to 7334736174053404672. Current Snapshot: MvccSnapshot[applied={T|T < 7334736174393544704}]
I20260929 18:37:53.830802 21162 sys_catalog.cc:455] T 00000000000000000000000000000000 P cdfc2d79001d4a34ae6eddb06d8a0c0d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "cdfc2d79001d4a34ae6eddb06d8a0c0d" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "16cac5b4dfe74684861c10d815fb051c" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 34955 } } peers { permanent_uuid: "30e554990e1d47fb8e2b183647f95d5d" member_type: VOTER last_known_addr { host: "127.13.100.61" port: 46389 } attrs { promote: false } } peers { permanent_uuid: "cdfc2d79001d4a34ae6eddb06d8a0c0d" member_type: VOTER last_known_addr { host: "127.13.100.60" port: 40697 } attrs { promote: false } } }
I20260929 18:37:53.830924 21162 sys_catalog.cc:458] T 00000000000000000000000000000000 P cdfc2d79001d4a34ae6eddb06d8a0c0d [sys.catalog]: This master's current role is: LEADER
I20260929 18:37:53.831487 21169 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30e554990e1d47fb8e2b183647f95d5d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "cdfc2d79001d4a34ae6eddb06d8a0c0d" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "16cac5b4dfe74684861c10d815fb051c" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 34955 } } peers { permanent_uuid: "30e554990e1d47fb8e2b183647f95d5d" member_type: VOTER last_known_addr { host: "127.13.100.61" port: 46389 } attrs { promote: false } } peers { permanent_uuid: "cdfc2d79001d4a34ae6eddb06d8a0c0d" member_type: VOTER last_known_addr { host: "127.13.100.60" port: 40697 } attrs { promote: false } } }
I20260929 18:37:53.833228 21169 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30e554990e1d47fb8e2b183647f95d5d [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:37:53.841996 21178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:53.837504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43394 (local address 127.13.100.1:38831)
0929 18:37:53.837829 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:53.837835 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:53.837929 (+ 94us) server_negotiation.cc:407] Connection header received
0929 18:37:53.838053 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:53.838057 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:53.838105 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:53.838199 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:37:53.838407 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:53.839124 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:53.840847 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:53.841395 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:53.841513 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:53.841578 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:53.841769 (+ 191us) server_negotiation.cc:299] Negotiation successful
0929 18:37:53.841839 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":56,"threads_started":1}
I20260929 18:37:53.841996 21163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:53.836863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.3:35631 (local address 127.0.0.1:33144)
0929 18:37:53.836956 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:53.836976 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:53.837074 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:53.838422 (+ 1348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:53.838426 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:53.838443 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:37:53.838513 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:53.838516 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:53.839372 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:53.839378 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:53.840662 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:53.840670 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:53.841524 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:53.841528 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:53.841653 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:53.841679 (+ 26us) client_negotiation.cc:769] Sending connection context
0929 18:37:53.841723 (+ 44us) client_negotiation.cc:240] Negotiation successful
0929 18:37:53.841785 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28,"mutex_wait_us":265}
I20260929 18:37:53.842226 21174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:53.836963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33144 (local address 127.13.100.3:35631)
0929 18:37:53.838188 (+ 1225us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:53.838192 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:53.838206 (+ 14us) server_negotiation.cc:407] Connection header received
0929 18:37:53.838252 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:53.838255 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:53.838301 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:53.838401 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:37:53.838619 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:53.839270 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:53.840774 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:53.841395 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:53.841931 (+ 536us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:53.841990 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:53.842024 (+ 34us) server_negotiation.cc:299] Negotiation successful
0929 18:37:53.842091 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1133,"thread_start_us":62,"threads_started":1}
I20260929 18:37:53.842571 21176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:53.837387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.1:38831 (local address 127.0.0.1:43394)
0929 18:37:53.837825 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:53.837839 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:53.837965 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:53.838207 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:53.838209 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:53.838221 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:37:53.838302 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:53.838306 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:53.839263 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:53.839266 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:53.840729 (+ 1463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:53.840737 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:53.841511 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:53.841515 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:53.841639 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:53.841667 (+ 28us) client_negotiation.cc:769] Sending connection context
0929 18:37:53.841755 (+ 88us) client_negotiation.cc:240] Negotiation successful
0929 18:37:53.841825 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":379,"mutex_wait_us":69,"thread_start_us":95,"threads_started":1}
I20260929 18:37:53.842788 21007 tablet_service.cc:1467] Processing CreateTablet for tablet 87c4f664706147539dc49132dcfc0a43 (DEFAULT_TABLE table=test-table [id=4cca52a88e394c0989d8ed0484622639]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:37:53.842886 21175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:53.837352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.2:45533 (local address 127.0.0.1:34936)
0929 18:37:53.837974 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:53.837990 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:53.838066 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:53.838835 (+ 769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:53.838838 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:53.838855 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:37:53.838946 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:53.838951 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:53.839923 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:53.839926 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:53.841554 (+ 1628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:53.841562 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:53.842516 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:53.842518 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:53.842618 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:53.842638 (+ 20us) client_negotiation.cc:769] Sending connection context
0929 18:37:53.842705 (+ 67us) client_negotiation.cc:240] Negotiation successful
0929 18:37:53.842763 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"mutex_wait_us":480,"thread_start_us":103,"threads_started":1}
I20260929 18:37:53.843112 21007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87c4f664706147539dc49132dcfc0a43. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:37:53.843134 20734 tablet_service.cc:1467] Processing CreateTablet for tablet 87c4f664706147539dc49132dcfc0a43 (DEFAULT_TABLE table=test-table [id=4cca52a88e394c0989d8ed0484622639]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:37:53.843259 21177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:53.837428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34936 (local address 127.13.100.2:45533)
0929 18:37:53.838607 (+ 1179us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:53.838611 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:53.838624 (+ 13us) server_negotiation.cc:407] Connection header received
0929 18:37:53.838669 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:53.838671 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:53.838725 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:53.838807 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:37:53.839060 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:53.839793 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:53.841677 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:53.842271 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:53.842393 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:53.842456 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:53.843064 (+ 608us) server_negotiation.cc:299] Negotiation successful
0929 18:37:53.843146 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1101,"thread_start_us":60,"threads_started":1}
I20260929 18:37:53.843330 20734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87c4f664706147539dc49132dcfc0a43. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:37:53.843741 20871 tablet_service.cc:1467] Processing CreateTablet for tablet 87c4f664706147539dc49132dcfc0a43 (DEFAULT_TABLE table=test-table [id=4cca52a88e394c0989d8ed0484622639]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:37:53.843961 20871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87c4f664706147539dc49132dcfc0a43. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:37:53.845805 21180 tablet_bootstrap.cc:492] T 87c4f664706147539dc49132dcfc0a43 P 8600dd95d7d441f9aaf8fc399cfccb14: Bootstrap starting.
I20260929 18:37:53.846246 21179 tablet_bootstrap.cc:492] T 87c4f664706147539dc49132dcfc0a43 P 2fbaf12215ba453bb80f9cccaaff2e73: Bootstrap starting.
I20260929 18:37:53.846792 21180 tablet_bootstrap.cc:654] T 87c4f664706147539dc49132dcfc0a43 P 8600dd95d7d441f9aaf8fc399cfccb14: Neither blocks nor log segments found. Creating new log.
I20260929 18:37:53.846926 21181 tablet_bootstrap.cc:492] T 87c4f664706147539dc49132dcfc0a43 P a579500ca64f489886008b02eba510bf: Bootstrap starting.
I20260929 18:37:53.847190 21179 tablet_bootstrap.cc:654] T 87c4f664706147539dc49132dcfc0a43 P 2fbaf12215ba453bb80f9cccaaff2e73: Neither blocks nor log segments found. Creating new log.
I20260929 18:37:53.847970 21181 tablet_bootstrap.cc:654] T 87c4f664706147539dc49132dcfc0a43 P a579500ca64f489886008b02eba510bf: Neither blocks nor log segments found. Creating new log.
I20260929 18:37:53.848100 21180 tablet_bootstrap.cc:492] T 87c4f664706147539dc49132dcfc0a43 P 8600dd95d7d441f9aaf8fc399cfccb14: No bootstrap required, opened a new log
I20260929 18:37:53.848191 21180 ts_tablet_manager.cc:1397] T 87c4f664706147539dc49132dcfc0a43 P 8600dd95d7d441f9aaf8fc399cfccb14: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260929 18:37:53.848495 21179 tablet_bootstrap.cc:492] T 87c4f664706147539dc49132dcfc0a43 P 2fbaf12215ba453bb80f9cccaaff2e73: No bootstrap required, opened a new log
I20260929 18:37:53.848608 21179 ts_tablet_manager.cc:1397] T 87c4f664706147539dc49132dcfc0a43 P 2fbaf12215ba453bb80f9cccaaff2e73: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 18:37:53.848729 21180 raft_consensus.cc:348] T 87c4f664706147539dc49132dcfc0a43 P 8600dd95d7d441f9aaf8fc399cfccb14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fbaf12215ba453bb80f9cccaaff2e73" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 35631 } } peers { permanent_uuid: "8600dd95d7d441f9aaf8fc399cfccb14" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 38831 } } peers { permanent_uuid: "a579500ca64f489886008b02eba510bf" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 45533 } }
I20260929 18:37:53.848868 21180 raft_consensus.cc:374] T 87c4f664706147539dc49132dcfc0a43 P 8600dd95d7d441f9aaf8fc399cfccb14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:37:53.848907 21180 raft_consensus.cc:729] T 87c4f664706147539dc49132dcfc0a43 P 8600dd95d7d441f9aaf8fc399cfccb14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8600dd95d7d441f9aaf8fc399cfccb14, State: Initialized, Role: FOLLOWER
I20260929 18:37:53.849051 21180 consensus_queue.cc:260] T 87c4f664706147539dc49132dcfc0a43 P 8600dd95d7d441f9aaf8fc399cfccb14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fbaf12215ba453bb80f9cccaaff2e73" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 35631 } } peers { permanent_uuid: "8600dd95d7d441f9aaf8fc399cfccb14" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 38831 } } peers { permanent_uuid: "a579500ca64f489886008b02eba510bf" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 45533 } }
I20260929 18:37:53.849105 21181 tablet_bootstrap.cc:492] T 87c4f664706147539dc49132dcfc0a43 P a579500ca64f489886008b02eba510bf: No bootstrap required, opened a new log
I20260929 18:37:53.849056 21179 raft_consensus.cc:348] T 87c4f664706147539dc49132dcfc0a43 P 2fbaf12215ba453bb80f9cccaaff2e73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fbaf12215ba453bb80f9cccaaff2e73" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 35631 } } peers { permanent_uuid: "8600dd95d7d441f9aaf8fc399cfccb14" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 38831 } } peers { permanent_uuid: "a579500ca64f489886008b02eba510bf" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 45533 } }
I20260929 18:37:53.849160 21179 raft_consensus.cc:374] T 87c4f664706147539dc49132dcfc0a43 P 2fbaf12215ba453bb80f9cccaaff2e73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:37:53.849174 21181 ts_tablet_manager.cc:1397] T 87c4f664706147539dc49132dcfc0a43 P a579500ca64f489886008b02eba510bf: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 18:37:53.849234 21179 raft_consensus.cc:729] T 87c4f664706147539dc49132dcfc0a43 P 2fbaf12215ba453bb80f9cccaaff2e73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fbaf12215ba453bb80f9cccaaff2e73, State: Initialized, Role: FOLLOWER
I20260929 18:37:53.849442 21179 consensus_queue.cc:260] T 87c4f664706147539dc49132dcfc0a43 P 2fbaf12215ba453bb80f9cccaaff2e73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fbaf12215ba453bb80f9cccaaff2e73" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 35631 } } peers { permanent_uuid: "8600dd95d7d441f9aaf8fc399cfccb14" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 38831 } } peers { permanent_uuid: "a579500ca64f489886008b02eba510bf" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 45533 } }
I20260929 18:37:53.849864 21179 ts_tablet_manager.cc:1428] T 87c4f664706147539dc49132dcfc0a43 P 2fbaf12215ba453bb80f9cccaaff2e73: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 18:37:53.849959 21159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:53.849699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:34955 (local address 127.13.100.1:53045)
0929 18:37:53.849795 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:53.849869 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.62:34955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260929 18:37:53.850075 21181 raft_consensus.cc:348] T 87c4f664706147539dc49132dcfc0a43 P a579500ca64f489886008b02eba510bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fbaf12215ba453bb80f9cccaaff2e73" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 35631 } } peers { permanent_uuid: "8600dd95d7d441f9aaf8fc399cfccb14" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 38831 } } peers { permanent_uuid: "a579500ca64f489886008b02eba510bf" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 45533 } }
I20260929 18:37:53.850302 21181 raft_consensus.cc:374] T 87c4f664706147539dc49132dcfc0a43 P a579500ca64f489886008b02eba510bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:37:53.850381 21181 raft_consensus.cc:729] T 87c4f664706147539dc49132dcfc0a43 P a579500ca64f489886008b02eba510bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a579500ca64f489886008b02eba510bf, State: Initialized, Role: FOLLOWER
I20260929 18:37:53.851112 21181 consensus_queue.cc:260] T 87c4f664706147539dc49132dcfc0a43 P a579500ca64f489886008b02eba510bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fbaf12215ba453bb80f9cccaaff2e73" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 35631 } } peers { permanent_uuid: "8600dd95d7d441f9aaf8fc399cfccb14" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 38831 } } peers { permanent_uuid: "a579500ca64f489886008b02eba510bf" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 45533 } }
I20260929 18:37:53.851485 21181 ts_tablet_manager.cc:1428] T 87c4f664706147539dc49132dcfc0a43 P a579500ca64f489886008b02eba510bf: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260929 18:37:53.852067 21161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:53.851816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:34955 (local address 127.13.100.2:42035)
0929 18:37:53.851949 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:53.852005 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.62:34955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260929 18:37:53.852567 21160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:53.850652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:34955 (local address 127.13.100.3:38879)
0929 18:37:53.852434 (+ 1782us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:53.852503 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.62:34955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1709}
I20260929 18:37:53.852684 21074 heartbeater.cc:503] Master 127.13.100.60:40697 was elected leader, sending a full tablet report...
I20260929 18:37:53.853361 20937 heartbeater.cc:503] Master 127.13.100.60:40697 was elected leader, sending a full tablet report...
I20260929 18:37:53.853513 21160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:53.853064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:34955 (local address 127.13.100.3:55965)
0929 18:37:53.853402 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:53.853442 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.62:34955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":271}
I20260929 18:37:53.853688 21161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:53.853462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:34955 (local address 127.13.100.2:58883)
0929 18:37:53.853582 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:53.853630 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.62:34955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260929 18:37:53.854163 21180 ts_tablet_manager.cc:1428] T 87c4f664706147539dc49132dcfc0a43 P 8600dd95d7d441f9aaf8fc399cfccb14: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20260929 18:37:53.854347 20800 heartbeater.cc:503] Master 127.13.100.60:40697 was elected leader, sending a full tablet report...
I20260929 18:37:53.854943 21159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:53.854690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:34955 (local address 127.13.100.1:53425)
0929 18:37:53.854826 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:53.854877 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.62:34955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
W20260929 18:37:54.073423 20940 tablet.cc:2369] T 87c4f664706147539dc49132dcfc0a43 P a579500ca64f489886008b02eba510bf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:37:54.096172 21185 raft_consensus.cc:482] T 87c4f664706147539dc49132dcfc0a43 P 8600dd95d7d441f9aaf8fc399cfccb14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:37:54.096345 21185 raft_consensus.cc:504] T 87c4f664706147539dc49132dcfc0a43 P 8600dd95d7d441f9aaf8fc399cfccb14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fbaf12215ba453bb80f9cccaaff2e73" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 35631 } } peers { permanent_uuid: "8600dd95d7d441f9aaf8fc399cfccb14" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 38831 } } peers { permanent_uuid: "a579500ca64f489886008b02eba510bf" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 45533 } }
I20260929 18:37:54.097565 21185 leader_election.cc:288] T 87c4f664706147539dc49132dcfc0a43 P 8600dd95d7d441f9aaf8fc399cfccb14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2fbaf12215ba453bb80f9cccaaff2e73 (127.13.100.3:35631), a579500ca64f489886008b02eba510bf (127.13.100.2:45533)
I20260929 18:37:54.102591 21174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:54.097980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.1:58411 (local address 127.13.100.3:35631)
0929 18:37:54.098121 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:54.098128 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:54.098152 (+ 24us) server_negotiation.cc:407] Connection header received
0929 18:37:54.098355 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:54.098360 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:54.098420 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:54.098504 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:37:54.098799 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:54.099615 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:54.101253 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:54.101844 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:54.102009 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:54.102061 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:54.102379 (+ 318us) server_negotiation.cc:299] Negotiation successful
0929 18:37:54.102453 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 18:37:54.102591 21177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:54.098295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.1:47557 (local address 127.13.100.2:45533)
0929 18:37:54.098426 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:54.098430 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:54.098650 (+ 220us) server_negotiation.cc:407] Connection header received
0929 18:37:54.098734 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:54.098757 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:54.098806 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:54.098880 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:37:54.099098 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:54.099794 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:54.101253 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:54.101845 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:54.102005 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:54.102061 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:54.102391 (+ 330us) server_negotiation.cc:299] Negotiation successful
0929 18:37:54.102460 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 18:37:54.102631 21159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:54.097862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.3:35631 (local address 127.13.100.1:58411)
0929 18:37:54.098068 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:54.098086 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:54.098217 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:54.098534 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:54.098537 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:54.098559 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:37:54.098683 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:54.098692 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:54.099727 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:54.099730 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:54.101123 (+ 1393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:54.101133 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:54.101995 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:54.101999 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:54.102227 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:54.102271 (+ 44us) client_negotiation.cc:769] Sending connection context
0929 18:37:54.102357 (+ 86us) client_negotiation.cc:240] Negotiation successful
0929 18:37:54.102464 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86,"mutex_wait_us":181}
I20260929 18:37:54.102639 21188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:54.098188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.2:45533 (local address 127.13.100.1:47557)
0929 18:37:54.098548 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:54.098560 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:54.098651 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:54.098898 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:54.098900 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:54.098914 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:37:54.098996 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:54.099000 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:54.099897 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:54.099900 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:54.101123 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:54.101133 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:54.102004 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:54.102007 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:54.102231 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:54.102271 (+ 40us) client_negotiation.cc:769] Sending connection context
0929 18:37:54.102353 (+ 82us) client_negotiation.cc:240] Negotiation successful
0929 18:37:54.102465 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"mutex_wait_us":78,"thread_start_us":183,"threads_started":1}
I20260929 18:37:54.103128 21027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87c4f664706147539dc49132dcfc0a43" candidate_uuid: "8600dd95d7d441f9aaf8fc399cfccb14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fbaf12215ba453bb80f9cccaaff2e73" is_pre_election: true
I20260929 18:37:54.103292 21027 raft_consensus.cc:2454] T 87c4f664706147539dc49132dcfc0a43 P 2fbaf12215ba453bb80f9cccaaff2e73 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8600dd95d7d441f9aaf8fc399cfccb14 in term 0.
I20260929 18:37:54.103437 20891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87c4f664706147539dc49132dcfc0a43" candidate_uuid: "8600dd95d7d441f9aaf8fc399cfccb14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a579500ca64f489886008b02eba510bf" is_pre_election: true
I20260929 18:37:54.103652 20687 leader_election.cc:302] T 87c4f664706147539dc49132dcfc0a43 P 8600dd95d7d441f9aaf8fc399cfccb14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fbaf12215ba453bb80f9cccaaff2e73, 8600dd95d7d441f9aaf8fc399cfccb14; no voters:
I20260929 18:37:54.103696 20891 raft_consensus.cc:2454] T 87c4f664706147539dc49132dcfc0a43 P a579500ca64f489886008b02eba510bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8600dd95d7d441f9aaf8fc399cfccb14 in term 0.
I20260929 18:37:54.103936 21185 raft_consensus.cc:2781] T 87c4f664706147539dc49132dcfc0a43 P 8600dd95d7d441f9aaf8fc399cfccb14 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:37:54.103991 21185 raft_consensus.cc:482] T 87c4f664706147539dc49132dcfc0a43 P 8600dd95d7d441f9aaf8fc399cfccb14 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:37:54.104051 21185 raft_consensus.cc:3037] T 87c4f664706147539dc49132dcfc0a43 P 8600dd95d7d441f9aaf8fc399cfccb14 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:37:54.105149 21185 raft_consensus.cc:504] T 87c4f664706147539dc49132dcfc0a43 P 8600dd95d7d441f9aaf8fc399cfccb14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fbaf12215ba453bb80f9cccaaff2e73" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 35631 } } peers { permanent_uuid: "8600dd95d7d441f9aaf8fc399cfccb14" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 38831 } } peers { permanent_uuid: "a579500ca64f489886008b02eba510bf" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 45533 } }
I20260929 18:37:54.105506 21185 leader_election.cc:288] T 87c4f664706147539dc49132dcfc0a43 P 8600dd95d7d441f9aaf8fc399cfccb14 [CANDIDATE]: Term 1 election: Requested vote from peers 2fbaf12215ba453bb80f9cccaaff2e73 (127.13.100.3:35631), a579500ca64f489886008b02eba510bf (127.13.100.2:45533)
I20260929 18:37:54.105818 21027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87c4f664706147539dc49132dcfc0a43" candidate_uuid: "8600dd95d7d441f9aaf8fc399cfccb14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fbaf12215ba453bb80f9cccaaff2e73"
I20260929 18:37:54.105909 20891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87c4f664706147539dc49132dcfc0a43" candidate_uuid: "8600dd95d7d441f9aaf8fc399cfccb14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a579500ca64f489886008b02eba510bf"
I20260929 18:37:54.105968 21027 raft_consensus.cc:3037] T 87c4f664706147539dc49132dcfc0a43 P 2fbaf12215ba453bb80f9cccaaff2e73 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:37:54.106029 20891 raft_consensus.cc:3037] T 87c4f664706147539dc49132dcfc0a43 P a579500ca64f489886008b02eba510bf [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:37:54.106949 20891 raft_consensus.cc:2454] T 87c4f664706147539dc49132dcfc0a43 P a579500ca64f489886008b02eba510bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8600dd95d7d441f9aaf8fc399cfccb14 in term 1.
I20260929 18:37:54.107012 21027 raft_consensus.cc:2454] T 87c4f664706147539dc49132dcfc0a43 P 2fbaf12215ba453bb80f9cccaaff2e73 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8600dd95d7d441f9aaf8fc399cfccb14 in term 1.
I20260929 18:37:54.107216 20687 leader_election.cc:302] T 87c4f664706147539dc49132dcfc0a43 P 8600dd95d7d441f9aaf8fc399cfccb14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8600dd95d7d441f9aaf8fc399cfccb14, a579500ca64f489886008b02eba510bf; no voters:
I20260929 18:37:54.107422 21185 raft_consensus.cc:2781] T 87c4f664706147539dc49132dcfc0a43 P 8600dd95d7d441f9aaf8fc399cfccb14 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:37:54.107658 21185 raft_consensus.cc:686] T 87c4f664706147539dc49132dcfc0a43 P 8600dd95d7d441f9aaf8fc399cfccb14 [term 1 LEADER]: Becoming Leader. State: Replica: 8600dd95d7d441f9aaf8fc399cfccb14, State: Running, Role: LEADER
I20260929 18:37:54.107921 21185 consensus_queue.cc:237] T 87c4f664706147539dc49132dcfc0a43 P 8600dd95d7d441f9aaf8fc399cfccb14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fbaf12215ba453bb80f9cccaaff2e73" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 35631 } } peers { permanent_uuid: "8600dd95d7d441f9aaf8fc399cfccb14" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 38831 } } peers { permanent_uuid: "a579500ca64f489886008b02eba510bf" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 45533 } }
I20260929 18:37:54.108383 21188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:54.108137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:34955 (local address 127.13.100.1:47047)
0929 18:37:54.108253 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:54.108310 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.62:34955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260929 18:37:54.110425 20620 catalog_manager.cc:5495] T 87c4f664706147539dc49132dcfc0a43 P 8600dd95d7d441f9aaf8fc399cfccb14 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8600dd95d7d441f9aaf8fc399cfccb14 (127.13.100.1). New cstate: current_term: 1 leader_uuid: "8600dd95d7d441f9aaf8fc399cfccb14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fbaf12215ba453bb80f9cccaaff2e73" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 35631 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8600dd95d7d441f9aaf8fc399cfccb14" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 38831 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a579500ca64f489886008b02eba510bf" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 45533 } health_report { overall_health: UNKNOWN } } }
I20260929 18:37:54.136778 21178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:54.133267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43396 (local address 127.13.100.1:38831)
0929 18:37:54.133406 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:54.133411 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:54.133427 (+ 16us) server_negotiation.cc:407] Connection header received
0929 18:37:54.133552 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:54.133554 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:54.133615 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:54.133736 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:37:54.134036 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:54.134539 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:54.135376 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:54.135593 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:54.135702 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:54.135966 (+ 264us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:37:54.136392 (+ 426us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:37:54.136514 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:54.136580 (+ 66us) server_negotiation.cc:299] Negotiation successful
0929 18:37:54.136633 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 18:37:54.139384 21027 raft_consensus.cc:1261] T 87c4f664706147539dc49132dcfc0a43 P 2fbaf12215ba453bb80f9cccaaff2e73 [term 1 FOLLOWER]: Refusing update from remote peer 8600dd95d7d441f9aaf8fc399cfccb14: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:37:54.139616 20891 raft_consensus.cc:1261] T 87c4f664706147539dc49132dcfc0a43 P a579500ca64f489886008b02eba510bf [term 1 FOLLOWER]: Refusing update from remote peer 8600dd95d7d441f9aaf8fc399cfccb14: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:37:54.139931 21185 consensus_queue.cc:1035] T 87c4f664706147539dc49132dcfc0a43 P 8600dd95d7d441f9aaf8fc399cfccb14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2fbaf12215ba453bb80f9cccaaff2e73" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 35631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:37:54.140190 21189 consensus_queue.cc:1035] T 87c4f664706147539dc49132dcfc0a43 P 8600dd95d7d441f9aaf8fc399cfccb14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a579500ca64f489886008b02eba510bf" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 45533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:37:54.142307 21161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:54.141998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:34955 (local address 127.13.100.2:55991)
0929 18:37:54.142157 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:54.142223 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.62:34955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260929 18:37:54.143738 21160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:54.143478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:34955 (local address 127.13.100.3:37965)
0929 18:37:54.143619 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:54.143672 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.62:34955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260929 18:37:54.145146 21188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:54.144867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:34955 (local address 127.13.100.1:57395)
0929 18:37:54.145002 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:54.145073 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.62:34955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260929 18:37:54.145773 21160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:54.145481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:34955 (local address 127.13.100.3:59039)
0929 18:37:54.145662 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:54.145714 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.62:34955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
I20260929 18:37:54.147181 21193 mvcc.cc:205] Tried to move back new op lower bound from 7334736175670165504 to 7334736175549104128. Current Snapshot: MvccSnapshot[applied={T|T < 7334736175670165504}]
I20260929 18:37:54.149236 21161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:54.148880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:34955 (local address 127.13.100.2:36493)
0929 18:37:54.149118 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:54.149177 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.62:34955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260929 18:37:54.150369 21188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:54.148109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:34955 (local address 127.13.100.1:33891)
0929 18:37:54.150248 (+ 2139us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:54.150303 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.62:34955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2079}
I20260929 18:37:54.153445 20618 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54512:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:37:54.153571 20618 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54512:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:37:54.157256 20618 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cdfc2d79001d4a34ae6eddb06d8a0c0d: Sending DeleteTablet for 3 replicas of tablet 87c4f664706147539dc49132dcfc0a43
I20260929 18:37:54.157837 21007 tablet_service.cc:1514] Processing DeleteTablet for tablet 87c4f664706147539dc49132dcfc0a43 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:37:54 UTC) from {username='oryx', principal='oryx/127.13.100.60@KRBTEST.COM'} at 127.0.0.1:33144
I20260929 18:37:54.157967 20871 tablet_service.cc:1514] Processing DeleteTablet for tablet 87c4f664706147539dc49132dcfc0a43 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:37:54 UTC) from {username='oryx', principal='oryx/127.13.100.60@KRBTEST.COM'} at 127.0.0.1:34936
I20260929 18:37:54.158120 20734 tablet_service.cc:1514] Processing DeleteTablet for tablet 87c4f664706147539dc49132dcfc0a43 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:37:54 UTC) from {username='oryx', principal='oryx/127.13.100.60@KRBTEST.COM'} at 127.0.0.1:43394
I20260929 18:37:54.158280 21202 tablet_replica.cc:330] T 87c4f664706147539dc49132dcfc0a43 P 2fbaf12215ba453bb80f9cccaaff2e73: stopping tablet replica
I20260929 18:37:54.158254 21203 tablet_replica.cc:330] T 87c4f664706147539dc49132dcfc0a43 P a579500ca64f489886008b02eba510bf: stopping tablet replica
I20260929 18:37:54.158399 21203 raft_consensus.cc:2229] T 87c4f664706147539dc49132dcfc0a43 P a579500ca64f489886008b02eba510bf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:37:54.158404 21202 raft_consensus.cc:2229] T 87c4f664706147539dc49132dcfc0a43 P 2fbaf12215ba453bb80f9cccaaff2e73 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:37:54.158517 21202 raft_consensus.cc:2258] T 87c4f664706147539dc49132dcfc0a43 P 2fbaf12215ba453bb80f9cccaaff2e73 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:37:54.158519 21203 raft_consensus.cc:2258] T 87c4f664706147539dc49132dcfc0a43 P a579500ca64f489886008b02eba510bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:37:54.159204 21202 ts_tablet_manager.cc:1905] T 87c4f664706147539dc49132dcfc0a43 P 2fbaf12215ba453bb80f9cccaaff2e73: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:37:54.159405 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 20583
I20260929 18:37:54.159750 21203 ts_tablet_manager.cc:1905] T 87c4f664706147539dc49132dcfc0a43 P a579500ca64f489886008b02eba510bf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:37:54.162407 21202 ts_tablet_manager.cc:1918] T 87c4f664706147539dc49132dcfc0a43 P 2fbaf12215ba453bb80f9cccaaff2e73: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:37:54.162478 21202 log.cc:1196] T 87c4f664706147539dc49132dcfc0a43 P 2fbaf12215ba453bb80f9cccaaff2e73: Deleting WAL directory at /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/ts-2/wal/wals/87c4f664706147539dc49132dcfc0a43
I20260929 18:37:54.162485 21203 ts_tablet_manager.cc:1918] T 87c4f664706147539dc49132dcfc0a43 P a579500ca64f489886008b02eba510bf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:37:54.162549 21203 log.cc:1196] T 87c4f664706147539dc49132dcfc0a43 P a579500ca64f489886008b02eba510bf: Deleting WAL directory at /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707036041078-13712-0/minicluster-data/ts-1/wal/wals/87c4f664706147539dc49132dcfc0a43
I20260929 18:37:54.162762 21202 ts_tablet_manager.cc:1939] T 87c4f664706147539dc49132dcfc0a43 P 2fbaf12215ba453bb80f9cccaaff2e73: Deleting consensus metadata
I20260929 18:37:54.162827 21203 ts_tablet_manager.cc:1939] T 87c4f664706147539dc49132dcfc0a43 P a579500ca64f489886008b02eba510bf: Deleting consensus metadata
I20260929 18:37:54.163514 20605 catalog_manager.cc:4841] TS a579500ca64f489886008b02eba510bf (127.13.100.2:45533): tablet 87c4f664706147539dc49132dcfc0a43 (table test-table [id=4cca52a88e394c0989d8ed0484622639]) successfully deleted
I20260929 18:37:54.163707 20605 catalog_manager.cc:4841] TS 2fbaf12215ba453bb80f9cccaaff2e73 (127.13.100.3:35631): tablet 87c4f664706147539dc49132dcfc0a43 (table test-table [id=4cca52a88e394c0989d8ed0484622639]) successfully deleted
I20260929 18:37:54.168228 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 20808
I20260929 18:37:54.168418 21175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:54.168131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.1:38831 (local address 127.0.0.1:43400)
0929 18:37:54.168257 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:54.168319 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.1:38831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260929 18:37:54.169155 20607 catalog_manager.cc:4584] TS 8600dd95d7d441f9aaf8fc399cfccb14 (127.13.100.1:38831): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 87c4f664706147539dc49132dcfc0a43: Network error: Client connection negotiation failed: client connection to 127.13.100.1:38831: connect: Connection refused (error 111)
I20260929 18:37:54.175230 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 20944
I20260929 18:37:54.181147 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 20301
I20260929 18:37:54.186805 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 20410
2026-09-29T18:37:54Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (7192 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707036041078-13712-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:37:54 dist-test-slave-j1xs krb5kdc[21207](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:37:54 dist-test-slave-j1xs krb5kdc[21207](info): set up 2 sockets
Sep 29 18:37:54 dist-test-slave-j1xs krb5kdc[21207](info): commencing operation
krb5kdc: starting...
W20260929 18:37:55.226816 13712 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707036041078-13712-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707036041078-13712-0/krb5kdc/test-user.keytab.
Sep 29 18:37:55 dist-test-slave-j1xs krb5kdc[21207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707075, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:37:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:37:55Z Disabled control of system clock
WARNING: no policy specified for oryx/127.13.100.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.100.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.100.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707036041078-13712-0/krb5kdc/oryx_127.13.100.62.keytab.
Entry for principal oryx/127.13.100.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707036041078-13712-0/krb5kdc/oryx_127.13.100.62.keytab.
I20260929 18:37:55.347210 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707036041078-13712-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707036041078-13712-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707036041078-13712-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.100.62:39731
--webserver_interface=127.13.100.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707036041078-13712-0/krb5kdc/oryx_127.13.100.62.keytab
--principal=oryx/127.13.100.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.100.20:42145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.100.62:39731
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707036041078-13712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707036041078-13712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707036041078-13712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:37:55.456431 21220 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:37:55.456681 21220 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:37:55.456722 21220 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:37:55.460795 21220 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:37:55.460862 21220 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 18:37:55.460879 21220 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:37:55.460897 21220 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:37:55.460914 21220 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:37:55.460935 21220 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:37:55.466398 21220 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.100.20:42145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707036041078-13712-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707036041078-13712-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.100.62:39731
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707036041078-13712-0/krb5kdc/oryx_127.13.100.62.keytab
--principal=oryx/127.13.100.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.100.62:39731
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.100.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21220
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707036041078-13712-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:37:55.467435 21220 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:37:55.468433 21220 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 18:37:55.477198 21220 server_base.cc:1034] running on GCE node
W20260929 18:37:55.477217 21225 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:37:55.477331 21228 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:37:55.477155 21226 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:37:55.477977 21220 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:37:55.479082 21220 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:37:55.480242 21220 hybrid_clock.cc:648] HybridClock initialized: now 1790707075480228 us; error 27 us; skew 500 ppm
Sep 29 18:37:55 dist-test-slave-j1xs krb5kdc[21207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707075, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.100.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:37:55.487205 21220 init.cc:377] Logged in from keytab as oryx/127.13.100.62@KRBTEST.COM (short username oryx)
I20260929 18:37:55.488097 21220 webserver.cc:460] Webserver started at http://127.13.100.62:33627/ using document root <none> and password file <none>
I20260929 18:37:55.488654 21220 fs_manager.cc:359] Metadata directory not provided
I20260929 18:37:55.488715 21220 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:37:55.488940 21220 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:37:55.491062 21220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707036041078-13712-0/minicluster-data/master-0/data/instance:
uuid: "73570d97b6164fdb8636f3f3c01b24e4"
format_stamp: "Formatted at 2026-09-29 18:37:55 on dist-test-slave-j1xs"
server_key: "9939b2b56a526409ac0b5c684eb253fe"
server_key_iv: "6a751614e65d577ad35fb6418b95313d"
server_key_version: "encryptionkey@0"
I20260929 18:37:55.491618 21220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707036041078-13712-0/minicluster-data/master-0/wal/instance:
uuid: "73570d97b6164fdb8636f3f3c01b24e4"
format_stamp: "Formatted at 2026-09-29 18:37:55 on dist-test-slave-j1xs"
server_key: "9939b2b56a526409ac0b5c684eb253fe"
server_key_iv: "6a751614e65d577ad35fb6418b95313d"
server_key_version: "encryptionkey@0"
I20260929 18:37:55.495203 21220 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 18:37:55.497543 21235 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:37:55.498888 21220 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:37:55.499037 21220 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707036041078-13712-0/minicluster-data/master-0/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707036041078-13712-0/minicluster-data/master-0/wal
uuid: "73570d97b6164fdb8636f3f3c01b24e4"
format_stamp: "Formatted at 2026-09-29 18:37:55 on dist-test-slave-j1xs"
server_key: "9939b2b56a526409ac0b5c684eb253fe"
server_key_iv: "6a751614e65d577ad35fb6418b95313d"
server_key_version: "encryptionkey@0"
I20260929 18:37:55.499145 21220 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707036041078-13712-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707036041078-13712-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707036041078-13712-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:37:55.516888 21220 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:37:55.520028 21220 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:37:55.520251 21220 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:37:55.528216 21220 rpc_server.cc:307] RPC server started. Bound to: 127.13.100.62:39731
I20260929 18:37:55.528239 21297 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.100.62:39731 every 8 connection(s)
I20260929 18:37:55.529165 21220 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
I20260929 18:37:55.531813 21298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:37:55.533646 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 21220
I20260929 18:37:55.533752 13712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707036041078-13712-0/minicluster-data/master-0/wal/instance
I20260929 18:37:55.534092 13712 external_mini_cluster.cc:1428] Setting key b313989f40784e2386217642649879d4
I20260929 18:37:55.538056 21298 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73570d97b6164fdb8636f3f3c01b24e4: Bootstrap starting.
I20260929 18:37:55.540184 21298 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 73570d97b6164fdb8636f3f3c01b24e4: Neither blocks nor log segments found. Creating new log.
I20260929 18:37:55.540905 21298 log.cc:824] T 00000000000000000000000000000000 P 73570d97b6164fdb8636f3f3c01b24e4: Log is configured to *not* fsync() on all Append() calls
Sep 29 18:37:55 dist-test-slave-j1xs krb5kdc[21207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707075, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.13.100.62@KRBTEST.COM
I20260929 18:37:55.542955 21298 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73570d97b6164fdb8636f3f3c01b24e4: No bootstrap required, opened a new log
I20260929 18:37:55.546119 21298 raft_consensus.cc:348] T 00000000000000000000000000000000 P 73570d97b6164fdb8636f3f3c01b24e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73570d97b6164fdb8636f3f3c01b24e4" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 39731 } }
I20260929 18:37:55.546371 21298 raft_consensus.cc:374] T 00000000000000000000000000000000 P 73570d97b6164fdb8636f3f3c01b24e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:37:55.546442 21298 raft_consensus.cc:729] T 00000000000000000000000000000000 P 73570d97b6164fdb8636f3f3c01b24e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73570d97b6164fdb8636f3f3c01b24e4, State: Initialized, Role: FOLLOWER
I20260929 18:37:55.546877 21298 consensus_queue.cc:260] T 00000000000000000000000000000000 P 73570d97b6164fdb8636f3f3c01b24e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73570d97b6164fdb8636f3f3c01b24e4" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 39731 } }
I20260929 18:37:55.547008 21298 raft_consensus.cc:388] T 00000000000000000000000000000000 P 73570d97b6164fdb8636f3f3c01b24e4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:37:55.547093 21298 raft_consensus.cc:482] T 00000000000000000000000000000000 P 73570d97b6164fdb8636f3f3c01b24e4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:37:55.547207 21298 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 73570d97b6164fdb8636f3f3c01b24e4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:37:55.548142 21298 raft_consensus.cc:504] T 00000000000000000000000000000000 P 73570d97b6164fdb8636f3f3c01b24e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73570d97b6164fdb8636f3f3c01b24e4" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 39731 } }
I20260929 18:37:55.548471 21298 leader_election.cc:302] T 00000000000000000000000000000000 P 73570d97b6164fdb8636f3f3c01b24e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 73570d97b6164fdb8636f3f3c01b24e4; no voters:
I20260929 18:37:55.548784 21298 leader_election.cc:288] T 00000000000000000000000000000000 P 73570d97b6164fdb8636f3f3c01b24e4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:37:55.548799 21301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:55.535502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59340 (local address 127.13.100.62:39731)
0929 18:37:55.535935 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:55.535949 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:55.535983 (+ 34us) server_negotiation.cc:407] Connection header received
0929 18:37:55.536587 (+ 604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:55.536603 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:55.536916 (+ 313us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:55.537236 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:55.538080 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:55.539076 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:55.539881 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:55.540386 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:55.540547 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:55.543294 (+ 2747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:55.543299 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:55.543311 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:37:55.543336 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:55.545618 (+ 2282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:55.546167 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:55.546178 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:55.546184 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:55.546287 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:55.546617 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:55.546620 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:55.546621 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:55.547103 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:55.547307 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:55.547723 (+ 416us) server_negotiation.cc:299] Negotiation successful
0929 18:37:55.547999 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":112,"threads_started":1}
I20260929 18:37:55.548870 21303 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 73570d97b6164fdb8636f3f3c01b24e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:37:55.549140 21303 raft_consensus.cc:686] T 00000000000000000000000000000000 P 73570d97b6164fdb8636f3f3c01b24e4 [term 1 LEADER]: Becoming Leader. State: Replica: 73570d97b6164fdb8636f3f3c01b24e4, State: Running, Role: LEADER
I20260929 18:37:55.549456 21303 consensus_queue.cc:237] T 00000000000000000000000000000000 P 73570d97b6164fdb8636f3f3c01b24e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73570d97b6164fdb8636f3f3c01b24e4" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 39731 } }
I20260929 18:37:55.550181 21298 sys_catalog.cc:564] T 00000000000000000000000000000000 P 73570d97b6164fdb8636f3f3c01b24e4 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:37:55.551191 21304 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73570d97b6164fdb8636f3f3c01b24e4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "73570d97b6164fdb8636f3f3c01b24e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73570d97b6164fdb8636f3f3c01b24e4" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 39731 } } }
I20260929 18:37:55.551309 21304 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73570d97b6164fdb8636f3f3c01b24e4 [sys.catalog]: This master's current role is: LEADER
I20260929 18:37:55.551681 21305 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73570d97b6164fdb8636f3f3c01b24e4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 73570d97b6164fdb8636f3f3c01b24e4. Latest consensus state: current_term: 1 leader_uuid: "73570d97b6164fdb8636f3f3c01b24e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73570d97b6164fdb8636f3f3c01b24e4" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 39731 } } }
I20260929 18:37:55.551815 21312 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:37:55.554000 21305 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73570d97b6164fdb8636f3f3c01b24e4 [sys.catalog]: This master's current role is: LEADER
I20260929 18:37:55.554751 21312 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:37:55.560298 21312 catalog_manager.cc:1347] Generated new cluster ID: 018f02c1085549e7a13dd6982210d95c
I20260929 18:37:55.560369 21312 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:37:55.600060 21312 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:37:55.601046 21312 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:37:55.608624 21312 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 73570d97b6164fdb8636f3f3c01b24e4: Generated new TSK 0
I20260929 18:37:55.609405 21312 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.13.100.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.100.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.100.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707036041078-13712-0/krb5kdc/oryx_127.13.100.1.keytab.
Entry for principal oryx/127.13.100.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707036041078-13712-0/krb5kdc/oryx_127.13.100.1.keytab.
I20260929 18:37:55.637991 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707036041078-13712-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707036041078-13712-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707036041078-13712-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707036041078-13712-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.100.1:0
--local_ip_for_outbound_sockets=127.13.100.1
--webserver_interface=127.13.100.1
--webserver_port=0
--tserver_master_addrs=127.13.100.62:39731
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707036041078-13712-0/krb5kdc/oryx_127.13.100.1.keytab
--principal=oryx/127.13.100.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.100.20:42145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707036041078-13712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707036041078-13712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707036041078-13712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:37:55.745168 21324 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:37:55.745478 21324 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:37:55.745563 21324 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:37:55.749680 21324 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:37:55.749765 21324 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:37:55.749832 21324 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 18:37:55.749933 21324 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.100.1
I20260929 18:37:55.755331 21324 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.100.20:42145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707036041078-13712-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707036041078-13712-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707036041078-13712-0/krb5kdc/oryx_127.13.100.1.keytab
--principal=oryx/127.13.100.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.100.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707036041078-13712-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.100.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.13.100.62:39731
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21324
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.100.1
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707036041078-13712-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:37:55.756383 21324 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:37:55.757381 21324 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:37:55.766626 21329 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:37:55.766667 21332 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:37:55.766626 21330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:37:55.766999 21324 server_base.cc:1034] running on GCE node
I20260929 18:37:55.767388 21324 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:37:55.768064 21324 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:37:55.769250 21324 hybrid_clock.cc:648] HybridClock initialized: now 1790707075769231 us; error 29 us; skew 500 ppm
Sep 29 18:37:55 dist-test-slave-j1xs krb5kdc[21207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707075, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.100.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:37:55.772121 21324 init.cc:377] Logged in from keytab as oryx/127.13.100.1@KRBTEST.COM (short username oryx)
I20260929 18:37:55.773006 21324 webserver.cc:460] Webserver started at http://127.13.100.1:37659/ using document root <none> and password file <none>
I20260929 18:37:55.773509 21324 fs_manager.cc:359] Metadata directory not provided
I20260929 18:37:55.773566 21324 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:37:55.773721 21324 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:37:55.775695 21324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707036041078-13712-0/minicluster-data/ts-0/data/instance:
uuid: "1004ea961b344f06b6a5033e2c069359"
format_stamp: "Formatted at 2026-09-29 18:37:55 on dist-test-slave-j1xs"
server_key: "4fffd81d3f831056b33320c3b91ce2f4"
server_key_iv: "3b10bb589a6819da99ee4daefb14516a"
server_key_version: "encryptionkey@0"
I20260929 18:37:55.776185 21324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707036041078-13712-0/minicluster-data/ts-0/wal/instance:
uuid: "1004ea961b344f06b6a5033e2c069359"
format_stamp: "Formatted at 2026-09-29 18:37:55 on dist-test-slave-j1xs"
server_key: "4fffd81d3f831056b33320c3b91ce2f4"
server_key_iv: "3b10bb589a6819da99ee4daefb14516a"
server_key_version: "encryptionkey@0"
I20260929 18:37:55.779888 21324 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 18:37:55.782215 21339 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:37:55.783370 21324 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260929 18:37:55.783493 21324 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707036041078-13712-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707036041078-13712-0/minicluster-data/ts-0/wal
uuid: "1004ea961b344f06b6a5033e2c069359"
format_stamp: "Formatted at 2026-09-29 18:37:55 on dist-test-slave-j1xs"
server_key: "4fffd81d3f831056b33320c3b91ce2f4"
server_key_iv: "3b10bb589a6819da99ee4daefb14516a"
server_key_version: "encryptionkey@0"
I20260929 18:37:55.783613 21324 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707036041078-13712-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707036041078-13712-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707036041078-13712-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:37:55.806457 21324 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:37:55.809227 21324 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:37:55.809414 21324 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:37:55.810833 21324 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:37:55.810905 21324 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:37:55.810976 21324 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:37:55.811021 21324 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 18:37:55 dist-test-slave-j1xs krb5kdc[21207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707075, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.100.1@KRBTEST.COM for oryx/127.13.100.62@KRBTEST.COM
I20260929 18:37:55.821916 21324 rpc_server.cc:307] RPC server started. Bound to: 127.13.100.1:45113
I20260929 18:37:55.821992 21454 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.100.1:45113 every 8 connection(s)
I20260929 18:37:55.822988 21324 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707036041078-13712-0/minicluster-data/ts-0/data/info.pb
I20260929 18:37:55.824433 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 21324
I20260929 18:37:55.824568 13712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707036041078-13712-0/minicluster-data/ts-0/wal/instance
I20260929 18:37:55.824723 21301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:55.811715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.1:48487 (local address 127.13.100.62:39731)
0929 18:37:55.811879 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:55.811882 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:55.812813 (+ 931us) server_negotiation.cc:407] Connection header received
0929 18:37:55.813958 (+ 1145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:55.813963 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:55.814027 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:55.814088 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:55.815647 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:55.816288 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:55.817472 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:55.817806 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:55.817943 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:55.820856 (+ 2913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:55.820859 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:55.820861 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:37:55.820883 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:55.822610 (+ 1727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:55.823235 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:55.823240 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:55.823243 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:55.823332 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:55.823707 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:55.823713 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:55.823715 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:55.823898 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:55.823974 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:55.824417 (+ 443us) server_negotiation.cc:299] Negotiation successful
0929 18:37:55.824556 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260929 18:37:55.824900 13712 external_mini_cluster.cc:1428] Setting key 65d5f23715a93a7c99190ae99336c8de
I20260929 18:37:55.825476 21348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:55.812195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:39731 (local address 127.13.100.1:48487)
0929 18:37:55.812662 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:55.812701 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:55.813700 (+ 999us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:55.814444 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:55.814450 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:55.814870 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:37:55.815506 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:55.815526 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:55.816412 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:55.816415 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:55.817350 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:55.817358 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:55.817997 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:55.818001 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:55.818162 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:55.818760 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:37:55.818787 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:37:55.820672 (+ 1885us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:37:55.822739 (+ 2067us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:55.822748 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:55.822764 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:55.823119 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:55.823424 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:55.823428 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:55.823431 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:55.823588 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:55.823986 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:37:55.823991 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:37:55.824193 (+ 202us) client_negotiation.cc:769] Sending connection context
0929 18:37:55.824341 (+ 148us) client_negotiation.cc:240] Negotiation successful
0929 18:37:55.824639 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":133,"threads_started":1}
I20260929 18:37:55.826840 21455 heartbeater.cc:347] Connected to a master server at 127.13.100.62:39731
I20260929 18:37:55.826969 21455 heartbeater.cc:464] Registering TS with master...
I20260929 18:37:55.827523 21455 heartbeater.cc:511] Master 127.13.100.62:39731 requested a full tablet report, sending...
I20260929 18:37:55.829150 21251 ts_manager.cc:194] Registered new tserver with Master: 1004ea961b344f06b6a5033e2c069359 (127.13.100.1:45113)
I20260929 18:37:55.830318 21251 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.13.100.1@KRBTEST.COM'} at 127.13.100.1:48487
WARNING: no policy specified for oryx/127.13.100.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.100.2@KRBTEST.COM" created.
I20260929 18:37:55.841178 21301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:55.828887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.1:55361 (local address 127.13.100.62:39731)
0929 18:37:55.829025 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:55.829031 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:55.834386 (+ 5355us) server_negotiation.cc:407] Connection header received
0929 18:37:55.834499 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:55.834503 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:55.834564 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:55.834664 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:55.835058 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:55.835651 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:55.836497 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:55.836743 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:55.836865 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:55.837880 (+ 1015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:55.837883 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:55.837905 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:37:55.837928 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:55.839587 (+ 1659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:55.839991 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:55.839994 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:55.839995 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:55.840091 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:55.840381 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:55.840384 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:55.840386 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:55.840558 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:55.840661 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:55.840909 (+ 248us) server_negotiation.cc:299] Negotiation successful
0929 18:37:55.841011 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 18:37:55.841192 21462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:55.833993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:39731 (local address 127.13.100.1:55361)
0929 18:37:55.834268 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:55.834280 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:55.834400 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:55.834675 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:55.834676 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:55.834835 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:37:55.834924 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:55.834929 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:55.835768 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:55.835770 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:55.836398 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:55.836406 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:55.836866 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:55.836868 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:55.836975 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:55.837448 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:37:55.837459 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:37:55.837750 (+ 291us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:37:55.839694 (+ 1944us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:55.839697 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:55.839698 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:55.839899 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:55.840191 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:55.840195 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:55.840197 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:55.840283 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:55.840661 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:37:55.840665 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:37:55.840753 (+ 88us) client_negotiation.cc:769] Sending connection context
0929 18:37:55.840858 (+ 105us) client_negotiation.cc:240] Negotiation successful
0929 18:37:55.841007 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":116,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.100.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707036041078-13712-0/krb5kdc/oryx_127.13.100.2.keytab.
Entry for principal oryx/127.13.100.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707036041078-13712-0/krb5kdc/oryx_127.13.100.2.keytab.
I20260929 18:37:55.853181 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707036041078-13712-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707036041078-13712-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707036041078-13712-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707036041078-13712-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.100.2:0
--local_ip_for_outbound_sockets=127.13.100.2
--webserver_interface=127.13.100.2
--webserver_port=0
--tserver_master_addrs=127.13.100.62:39731
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707036041078-13712-0/krb5kdc/oryx_127.13.100.2.keytab
--principal=oryx/127.13.100.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.100.20:42145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707036041078-13712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707036041078-13712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707036041078-13712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:37:55.961196 21464 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:37:55.961429 21464 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:37:55.961499 21464 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:37:55.965386 21464 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:37:55.965431 21464 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:37:55.965468 21464 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 18:37:55.965536 21464 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.100.2
I20260929 18:37:55.970530 21464 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.100.20:42145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707036041078-13712-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707036041078-13712-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707036041078-13712-0/krb5kdc/oryx_127.13.100.2.keytab
--principal=oryx/127.13.100.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.100.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707036041078-13712-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.100.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.13.100.62:39731
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21464
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.100.2
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707036041078-13712-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:37:55.971439 21464 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:37:55.972368 21464 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:37:55.981467 21470 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:37:55.981459 21472 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:37:55.981455 21469 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:37:55.981506 21464 server_base.cc:1034] running on GCE node
I20260929 18:37:55.982095 21464 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:37:55.982730 21464 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:37:55.983896 21464 hybrid_clock.cc:648] HybridClock initialized: now 1790707075983882 us; error 31 us; skew 500 ppm
Sep 29 18:37:55 dist-test-slave-j1xs krb5kdc[21207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707075, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.100.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:37:55.986511 21464 init.cc:377] Logged in from keytab as oryx/127.13.100.2@KRBTEST.COM (short username oryx)
I20260929 18:37:55.987377 21464 webserver.cc:460] Webserver started at http://127.13.100.2:43877/ using document root <none> and password file <none>
I20260929 18:37:55.987913 21464 fs_manager.cc:359] Metadata directory not provided
I20260929 18:37:55.987969 21464 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:37:55.988118 21464 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:37:55.990123 21464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707036041078-13712-0/minicluster-data/ts-1/data/instance:
uuid: "4dc6fe52db3c4e3983ca92bca03b4bbe"
format_stamp: "Formatted at 2026-09-29 18:37:55 on dist-test-slave-j1xs"
server_key: "751b7e42bd8ab9825c82b0bd92bbaecb"
server_key_iv: "f131af7424b0de8ec784d4e2f7a767da"
server_key_version: "encryptionkey@0"
I20260929 18:37:55.990758 21464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707036041078-13712-0/minicluster-data/ts-1/wal/instance:
uuid: "4dc6fe52db3c4e3983ca92bca03b4bbe"
format_stamp: "Formatted at 2026-09-29 18:37:55 on dist-test-slave-j1xs"
server_key: "751b7e42bd8ab9825c82b0bd92bbaecb"
server_key_iv: "f131af7424b0de8ec784d4e2f7a767da"
server_key_version: "encryptionkey@0"
I20260929 18:37:55.994293 21464 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:37:55.996587 21479 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:37:55.997831 21464 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:37:55.998015 21464 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707036041078-13712-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707036041078-13712-0/minicluster-data/ts-1/wal
uuid: "4dc6fe52db3c4e3983ca92bca03b4bbe"
format_stamp: "Formatted at 2026-09-29 18:37:55 on dist-test-slave-j1xs"
server_key: "751b7e42bd8ab9825c82b0bd92bbaecb"
server_key_iv: "f131af7424b0de8ec784d4e2f7a767da"
server_key_version: "encryptionkey@0"
I20260929 18:37:55.998114 21464 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707036041078-13712-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707036041078-13712-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707036041078-13712-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:37:56.010749 21464 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:37:56.013837 21464 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:37:56.014075 21464 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:37:56.015539 21464 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:37:56.015584 21464 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:37:56.015622 21464 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:37:56.015679 21464 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 18:37:56 dist-test-slave-j1xs krb5kdc[21207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707075, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.100.2@KRBTEST.COM for oryx/127.13.100.62@KRBTEST.COM
I20260929 18:37:56.026254 21464 rpc_server.cc:307] RPC server started. Bound to: 127.13.100.2:37145
I20260929 18:37:56.026575 21594 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.100.2:37145 every 8 connection(s)
I20260929 18:37:56.027415 21464 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707036041078-13712-0/minicluster-data/ts-1/data/info.pb
I20260929 18:37:56.029072 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 21464
I20260929 18:37:56.029201 13712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707036041078-13712-0/minicluster-data/ts-1/wal/instance
I20260929 18:37:56.029471 13712 external_mini_cluster.cc:1428] Setting key 5f31546897a093a876a89a97b89184e1
I20260929 18:37:56.030483 21301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:56.016405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.2:42697 (local address 127.13.100.62:39731)
0929 18:37:56.016524 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:56.016537 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:56.017523 (+ 986us) server_negotiation.cc:407] Connection header received
0929 18:37:56.018609 (+ 1086us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:56.018615 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:56.018698 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:56.018771 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:56.020442 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:56.021308 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:56.022978 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:56.023380 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:56.023498 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:56.026555 (+ 3057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:56.026559 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:56.026562 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:37:56.026591 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:56.028195 (+ 1604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:56.028813 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:56.028815 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:56.028817 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:56.028895 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:56.029318 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:56.029322 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:56.029323 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:56.029541 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:56.029639 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:56.030174 (+ 535us) server_negotiation.cc:299] Negotiation successful
0929 18:37:56.030319 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 18:37:56.031198 21488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:56.016919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:39731 (local address 127.13.100.2:42697)
0929 18:37:56.017387 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:56.017425 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:56.018332 (+ 907us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:56.019207 (+ 875us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:56.019219 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:56.019652 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:37:56.020303 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:56.020324 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:56.021431 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:56.021434 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:56.022838 (+ 1404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:56.022850 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:56.023533 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:56.023535 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:56.023672 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:56.024248 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:37:56.024268 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:37:56.026327 (+ 2059us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:37:56.028308 (+ 1981us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:56.028313 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:56.028326 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:56.028663 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:56.028997 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:56.029001 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:56.029003 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:56.029186 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:56.029636 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:37:56.029642 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:37:56.029879 (+ 237us) client_negotiation.cc:769] Sending connection context
0929 18:37:56.030113 (+ 234us) client_negotiation.cc:240] Negotiation successful
0929 18:37:56.030430 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":129,"threads_started":1}
I20260929 18:37:56.032469 21595 heartbeater.cc:347] Connected to a master server at 127.13.100.62:39731
I20260929 18:37:56.032634 21595 heartbeater.cc:464] Registering TS with master...
I20260929 18:37:56.033113 21595 heartbeater.cc:511] Master 127.13.100.62:39731 requested a full tablet report, sending...
I20260929 18:37:56.034897 21252 ts_manager.cc:194] Registered new tserver with Master: 4dc6fe52db3c4e3983ca92bca03b4bbe (127.13.100.2:37145)
I20260929 18:37:56.035764 21252 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.13.100.2@KRBTEST.COM'} at 127.13.100.2:42697
WARNING: no policy specified for oryx/127.13.100.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.100.3@KRBTEST.COM" created.
I20260929 18:37:56.043598 21602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:56.034873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:39731 (local address 127.13.100.2:34369)
0929 18:37:56.035270 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:56.035292 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:56.035400 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:56.035711 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:56.035713 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:56.035930 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:37:56.036041 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:56.036047 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:56.037308 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:56.037314 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:56.038356 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:56.038369 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:56.038829 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:56.038833 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:56.038951 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:56.039440 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:37:56.039460 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:37:56.039908 (+ 448us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:37:56.041733 (+ 1825us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:56.041759 (+ 26us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:56.041764 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:56.042092 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:56.042467 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:56.042470 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:56.042473 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:56.042597 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:56.043166 (+ 569us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:37:56.043169 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:37:56.043247 (+ 78us) client_negotiation.cc:769] Sending connection context
0929 18:37:56.043346 (+ 99us) client_negotiation.cc:240] Negotiation successful
0929 18:37:56.043452 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":250,"threads_started":1}
I20260929 18:37:56.043813 21301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:56.034909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.2:34369 (local address 127.13.100.62:39731)
0929 18:37:56.035010 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:56.035014 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:56.035361 (+ 347us) server_negotiation.cc:407] Connection header received
0929 18:37:56.035513 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:56.035516 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:56.035558 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:56.035608 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:56.036227 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:56.037139 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:56.038487 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:56.038697 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:56.038788 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:56.040049 (+ 1261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:56.040052 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:56.040054 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:37:56.040075 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:56.041604 (+ 1529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:56.042198 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:56.042201 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:56.042203 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:56.042287 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:56.042711 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:56.042716 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:56.042719 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:56.043012 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:56.043132 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:56.043435 (+ 303us) server_negotiation.cc:299] Negotiation successful
0929 18:37:56.043567 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.100.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707036041078-13712-0/krb5kdc/oryx_127.13.100.3.keytab.
Entry for principal oryx/127.13.100.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707036041078-13712-0/krb5kdc/oryx_127.13.100.3.keytab.
I20260929 18:37:56.058558 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707036041078-13712-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707036041078-13712-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707036041078-13712-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707036041078-13712-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.100.3:0
--local_ip_for_outbound_sockets=127.13.100.3
--webserver_interface=127.13.100.3
--webserver_port=0
--tserver_master_addrs=127.13.100.62:39731
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707036041078-13712-0/krb5kdc/oryx_127.13.100.3.keytab
--principal=oryx/127.13.100.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.100.20:42145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707036041078-13712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707036041078-13712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707036041078-13712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:37:56.166522 21604 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:37:56.166776 21604 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:37:56.166857 21604 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:37:56.170810 21604 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:37:56.170882 21604 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:37:56.170945 21604 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 18:37:56.171027 21604 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.100.3
I20260929 18:37:56.176050 21604 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.100.20:42145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707036041078-13712-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707036041078-13712-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707036041078-13712-0/krb5kdc/oryx_127.13.100.3.keytab
--principal=oryx/127.13.100.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.100.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707036041078-13712-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.100.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.13.100.62:39731
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21604
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.100.3
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707036041078-13712-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:37:56.176992 21604 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:37:56.177983 21604 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:37:56.187130 21609 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:37:56.187263 21612 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:37:56.187291 21610 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:37:56.187403 21604 server_base.cc:1034] running on GCE node
I20260929 18:37:56.187757 21604 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:37:56.188356 21604 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:37:56.189517 21604 hybrid_clock.cc:648] HybridClock initialized: now 1790707076189481 us; error 43 us; skew 500 ppm
Sep 29 18:37:56 dist-test-slave-j1xs krb5kdc[21207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707076, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.100.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:37:56.192106 21604 init.cc:377] Logged in from keytab as oryx/127.13.100.3@KRBTEST.COM (short username oryx)
I20260929 18:37:56.192850 21604 webserver.cc:460] Webserver started at http://127.13.100.3:42939/ using document root <none> and password file <none>
I20260929 18:37:56.193332 21604 fs_manager.cc:359] Metadata directory not provided
I20260929 18:37:56.193387 21604 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:37:56.193589 21604 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:37:56.195533 21604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707036041078-13712-0/minicluster-data/ts-2/data/instance:
uuid: "9741dfa235754052b0b0c2acc05483e6"
format_stamp: "Formatted at 2026-09-29 18:37:56 on dist-test-slave-j1xs"
server_key: "3c58ed5ba58179a0ae91219add5cd5ec"
server_key_iv: "97fd51c2f27141badd7f098994f1207e"
server_key_version: "encryptionkey@0"
I20260929 18:37:56.196076 21604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707036041078-13712-0/minicluster-data/ts-2/wal/instance:
uuid: "9741dfa235754052b0b0c2acc05483e6"
format_stamp: "Formatted at 2026-09-29 18:37:56 on dist-test-slave-j1xs"
server_key: "3c58ed5ba58179a0ae91219add5cd5ec"
server_key_iv: "97fd51c2f27141badd7f098994f1207e"
server_key_version: "encryptionkey@0"
I20260929 18:37:56.199640 21604 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 18:37:56.202137 21619 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:37:56.203460 21604 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:37:56.203593 21604 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707036041078-13712-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707036041078-13712-0/minicluster-data/ts-2/wal
uuid: "9741dfa235754052b0b0c2acc05483e6"
format_stamp: "Formatted at 2026-09-29 18:37:56 on dist-test-slave-j1xs"
server_key: "3c58ed5ba58179a0ae91219add5cd5ec"
server_key_iv: "97fd51c2f27141badd7f098994f1207e"
server_key_version: "encryptionkey@0"
I20260929 18:37:56.203717 21604 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707036041078-13712-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707036041078-13712-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707036041078-13712-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:37:56.224227 21604 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:37:56.227200 21604 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:37:56.227402 21604 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:37:56.228762 21604 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:37:56.228837 21604 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:37:56.228902 21604 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:37:56.228946 21604 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 18:37:56 dist-test-slave-j1xs krb5kdc[21207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707076, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.100.3@KRBTEST.COM for oryx/127.13.100.62@KRBTEST.COM
I20260929 18:37:56.239913 21604 rpc_server.cc:307] RPC server started. Bound to: 127.13.100.3:44055
I20260929 18:37:56.239984 21734 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.100.3:44055 every 8 connection(s)
I20260929 18:37:56.240747 21604 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707036041078-13712-0/minicluster-data/ts-2/data/info.pb
I20260929 18:37:56.242460 21301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:56.229657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.3:36093 (local address 127.13.100.62:39731)
0929 18:37:56.229786 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:56.229790 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:56.230852 (+ 1062us) server_negotiation.cc:407] Connection header received
0929 18:37:56.231903 (+ 1051us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:56.231907 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:56.231963 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:56.232048 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:56.233311 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:56.234141 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:56.235430 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:56.235779 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:56.235918 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:56.238762 (+ 2844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:56.238766 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:56.238768 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:37:56.238794 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:56.240372 (+ 1578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:56.240847 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:56.240852 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:56.240854 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:56.240932 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:56.241306 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:56.241309 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:56.241311 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:56.241545 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:56.241631 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:56.242238 (+ 607us) server_negotiation.cc:299] Negotiation successful
0929 18:37:56.242333 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 18:37:56.243280 21628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:56.230244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:39731 (local address 127.13.100.3:36093)
0929 18:37:56.230702 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:56.230740 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:56.231628 (+ 888us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:56.232227 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:56.232233 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:56.232594 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:37:56.233163 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:56.233177 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:56.234285 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:56.234287 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:56.235309 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:56.235318 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:56.235967 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:56.235971 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:56.236129 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:56.236727 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:37:56.236752 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:37:56.238607 (+ 1855us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:37:56.240483 (+ 1876us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:56.240488 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:56.240501 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:56.240743 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:56.241024 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:56.241028 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:56.241030 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:56.241190 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:56.241647 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:37:56.241653 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:37:56.241962 (+ 309us) client_negotiation.cc:769] Sending connection context
0929 18:37:56.242195 (+ 233us) client_negotiation.cc:240] Negotiation successful
0929 18:37:56.242575 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":134,"threads_started":1}
I20260929 18:37:56.244307 21735 heartbeater.cc:347] Connected to a master server at 127.13.100.62:39731
I20260929 18:37:56.244346 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 21604
I20260929 18:37:56.244483 13712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707036041078-13712-0/minicluster-data/ts-2/wal/instance
I20260929 18:37:56.244517 21735 heartbeater.cc:464] Registering TS with master...
I20260929 18:37:56.244784 13712 external_mini_cluster.cc:1428] Setting key 1672c7718fab538a84bb0bb0f776ffc6
I20260929 18:37:56.245069 21735 heartbeater.cc:511] Master 127.13.100.62:39731 requested a full tablet report, sending...
I20260929 18:37:56.246111 21252 ts_manager.cc:194] Registered new tserver with Master: 9741dfa235754052b0b0c2acc05483e6 (127.13.100.3:44055)
I20260929 18:37:56.246820 21252 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.13.100.3@KRBTEST.COM'} at 127.13.100.3:36093
I20260929 18:37:56.253364 21301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:56.246631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.3:58145 (local address 127.13.100.62:39731)
0929 18:37:56.246721 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:56.246725 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:56.246908 (+ 183us) server_negotiation.cc:407] Connection header received
0929 18:37:56.247014 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:56.247016 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:56.247060 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:56.247167 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:56.247563 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:56.248043 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:56.248824 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:56.249040 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:56.249146 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:56.250207 (+ 1061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:56.250210 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:56.250212 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:37:56.250263 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:56.251849 (+ 1586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:56.252242 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:56.252247 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:56.252248 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:56.252326 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:56.252604 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:56.252607 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:56.252609 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:56.252768 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:56.252871 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:56.253110 (+ 239us) server_negotiation.cc:299] Negotiation successful
0929 18:37:56.253207 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260929 18:37:56.253371 21741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:56.246573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:39731 (local address 127.13.100.3:58145)
0929 18:37:56.246819 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:56.246839 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:56.246937 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:56.247183 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:56.247186 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:56.247370 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:37:56.247456 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:56.247461 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:56.248162 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:56.248165 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:56.248714 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:56.248721 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:56.249147 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:56.249151 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:56.249236 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:56.249694 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:37:56.249704 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:37:56.250088 (+ 384us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:37:56.251954 (+ 1866us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:56.251958 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:56.251960 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:56.252148 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:56.252428 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:56.252431 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:56.252433 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:56.252514 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:56.252865 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:37:56.252868 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:37:56.252957 (+ 89us) client_negotiation.cc:769] Sending connection context
0929 18:37:56.253060 (+ 103us) client_negotiation.cc:240] Negotiation successful
0929 18:37:56.253194 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":88,"threads_started":1}
I20260929 18:37:56.257997 13712 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 18:37:56 dist-test-slave-j1xs krb5kdc[21207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707076, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 18:37:56 dist-test-slave-j1xs krb5kdc[21207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.13.100.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:37:56 dist-test-slave-j1xs krb5kdc[21207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.13.100.62@KRBTEST.COM, Server not found in Kerberos database
W20260929 18:37:56.281167 21301 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:37:56.274780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59356 (local address 127.13.100.62:39731)
0929 18:37:56.274903 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:56.274906 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:56.274920 (+ 14us) server_negotiation.cc:407] Connection header received
0929 18:37:56.274959 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:56.274961 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:56.274998 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:56.275070 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:56.275750 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:56.276258 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:56.277092 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:56.277307 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:56.277385 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:56.280966 (+ 3581us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59356: BlockingRecv error: recv got EOF from 127.0.0.1:59356 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 29 18:37:56 dist-test-slave-j1xs krb5kdc[21207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707076, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.13.100.62@KRBTEST.COM
I20260929 18:37:56.290783 21301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:56.282825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59372 (local address 127.13.100.62:39731)
0929 18:37:56.282946 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:56.282950 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:56.283062 (+ 112us) server_negotiation.cc:407] Connection header received
0929 18:37:56.283151 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:56.283153 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:56.283190 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:56.283262 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:56.283807 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:56.284291 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:56.285013 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:56.285209 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:56.285286 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:56.287621 (+ 2335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:56.287630 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:56.287632 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:37:56.287653 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:56.289095 (+ 1442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:56.289535 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:56.289540 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:56.289542 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:56.289621 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:56.289938 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:56.289944 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:56.289947 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:56.290124 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:56.290264 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:56.290510 (+ 246us) server_negotiation.cc:299] Negotiation successful
0929 18:37:56.290623 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 18:37:56.293650 21252 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59372:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:37:56.296651 21252 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:37:56.311657 21759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:56.305254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50074 (local address 127.13.100.2:37145)
0929 18:37:56.305605 (+ 351us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:56.305621 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:56.306252 (+ 631us) server_negotiation.cc:407] Connection header received
0929 18:37:56.306331 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:56.306336 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:56.306547 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:56.306700 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:37:56.307223 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:56.308575 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:56.310431 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:56.310967 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:56.311094 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:56.311264 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:56.311391 (+ 127us) server_negotiation.cc:299] Negotiation successful
0929 18:37:56.311489 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237,"thread_start_us":111,"threads_started":1}
I20260929 18:37:56.311657 21758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:56.305043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.2:37145 (local address 127.0.0.1:50074)
0929 18:37:56.305931 (+ 888us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:56.305980 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:56.306115 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:56.306870 (+ 755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:56.306875 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:56.306898 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:37:56.307094 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:56.307102 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:56.308702 (+ 1600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:56.308706 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:56.310290 (+ 1584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:56.310301 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:56.311094 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:56.311098 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:56.311202 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:56.311236 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 18:37:56.311356 (+ 120us) client_negotiation.cc:240] Negotiation successful
0929 18:37:56.311499 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":644,"mutex_wait_us":251,"thread_start_us":64,"threads_started":1}
I20260929 18:37:56.311695 21757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:56.305212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.1:45113 (local address 127.0.0.1:39396)
0929 18:37:56.306049 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:56.306064 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:56.306159 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:56.307146 (+ 987us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:56.307152 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:56.307171 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:37:56.307276 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:56.307282 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:56.308476 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:56.308481 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:56.310072 (+ 1591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:56.310082 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:56.311169 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:56.311172 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:56.311268 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:56.311290 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 18:37:56.311349 (+ 59us) client_negotiation.cc:240] Negotiation successful
0929 18:37:56.311514 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":767,"mutex_wait_us":43,"thread_start_us":126,"threads_started":1}
I20260929 18:37:56.312371 21761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:56.305761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39396 (local address 127.13.100.1:45113)
0929 18:37:56.306604 (+ 843us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:56.306621 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:56.306646 (+ 25us) server_negotiation.cc:407] Connection header received
0929 18:37:56.306714 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:56.306718 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:56.306885 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:56.307021 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:37:56.307390 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:56.308356 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:56.310193 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:56.310970 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:56.311923 (+ 953us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:56.312085 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:56.312134 (+ 49us) server_negotiation.cc:299] Negotiation successful
0929 18:37:56.312229 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":742,"thread_start_us":85,"threads_started":1}
I20260929 18:37:56.312395 21762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:56.305878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48708 (local address 127.13.100.3:44055)
0929 18:37:56.306318 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:56.306327 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:56.306712 (+ 385us) server_negotiation.cc:407] Connection header received
0929 18:37:56.306796 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:56.306799 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:56.307010 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:56.307164 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:37:56.307429 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:56.308625 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:56.310412 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:56.311164 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:56.311256 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:56.311413 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:56.311769 (+ 356us) server_negotiation.cc:299] Negotiation successful
0929 18:37:56.311867 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":101,"threads_started":1}
I20260929 18:37:56.312681 21760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:56.305716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.3:44055 (local address 127.0.0.1:48708)
0929 18:37:56.306474 (+ 758us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:56.306488 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:56.306602 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:56.307157 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:56.307161 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:56.307179 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:37:56.307300 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:56.307310 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:56.308740 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:56.308743 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:56.310295 (+ 1552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:56.310301 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:56.311469 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:56.311472 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:56.311552 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:56.311572 (+ 20us) client_negotiation.cc:769] Sending connection context
0929 18:37:56.311640 (+ 68us) client_negotiation.cc:240] Negotiation successful
0929 18:37:56.311696 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":665,"mutex_wait_us":292,"thread_start_us":112,"threads_started":1}
I20260929 18:37:56.315150 21389 tablet_service.cc:1467] Processing CreateTablet for tablet 8c1d8d25b91c456cbf72d6bb0c3869d7 (DEFAULT_TABLE table=test-table [id=3325c13033f74e66b10b375be4e5e09d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:37:56.315150 21529 tablet_service.cc:1467] Processing CreateTablet for tablet 8c1d8d25b91c456cbf72d6bb0c3869d7 (DEFAULT_TABLE table=test-table [id=3325c13033f74e66b10b375be4e5e09d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:37:56.315397 21669 tablet_service.cc:1467] Processing CreateTablet for tablet 8c1d8d25b91c456cbf72d6bb0c3869d7 (DEFAULT_TABLE table=test-table [id=3325c13033f74e66b10b375be4e5e09d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:37:56.316226 21529 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c1d8d25b91c456cbf72d6bb0c3869d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:37:56.316226 21389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c1d8d25b91c456cbf72d6bb0c3869d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:37:56.316295 21669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c1d8d25b91c456cbf72d6bb0c3869d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:37:56.322638 21763 tablet_bootstrap.cc:492] T 8c1d8d25b91c456cbf72d6bb0c3869d7 P 9741dfa235754052b0b0c2acc05483e6: Bootstrap starting.
I20260929 18:37:56.324174 21765 tablet_bootstrap.cc:492] T 8c1d8d25b91c456cbf72d6bb0c3869d7 P 1004ea961b344f06b6a5033e2c069359: Bootstrap starting.
I20260929 18:37:56.324978 21763 tablet_bootstrap.cc:654] T 8c1d8d25b91c456cbf72d6bb0c3869d7 P 9741dfa235754052b0b0c2acc05483e6: Neither blocks nor log segments found. Creating new log.
I20260929 18:37:56.325680 21763 log.cc:824] T 8c1d8d25b91c456cbf72d6bb0c3869d7 P 9741dfa235754052b0b0c2acc05483e6: Log is configured to *not* fsync() on all Append() calls
I20260929 18:37:56.325937 21765 tablet_bootstrap.cc:654] T 8c1d8d25b91c456cbf72d6bb0c3869d7 P 1004ea961b344f06b6a5033e2c069359: Neither blocks nor log segments found. Creating new log.
I20260929 18:37:56.326298 21764 tablet_bootstrap.cc:492] T 8c1d8d25b91c456cbf72d6bb0c3869d7 P 4dc6fe52db3c4e3983ca92bca03b4bbe: Bootstrap starting.
I20260929 18:37:56.326824 21765 log.cc:824] T 8c1d8d25b91c456cbf72d6bb0c3869d7 P 1004ea961b344f06b6a5033e2c069359: Log is configured to *not* fsync() on all Append() calls
I20260929 18:37:56.327862 21763 tablet_bootstrap.cc:492] T 8c1d8d25b91c456cbf72d6bb0c3869d7 P 9741dfa235754052b0b0c2acc05483e6: No bootstrap required, opened a new log
I20260929 18:37:56.328138 21763 ts_tablet_manager.cc:1397] T 8c1d8d25b91c456cbf72d6bb0c3869d7 P 9741dfa235754052b0b0c2acc05483e6: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260929 18:37:56.328277 21764 tablet_bootstrap.cc:654] T 8c1d8d25b91c456cbf72d6bb0c3869d7 P 4dc6fe52db3c4e3983ca92bca03b4bbe: Neither blocks nor log segments found. Creating new log.
I20260929 18:37:56.328794 21765 tablet_bootstrap.cc:492] T 8c1d8d25b91c456cbf72d6bb0c3869d7 P 1004ea961b344f06b6a5033e2c069359: No bootstrap required, opened a new log
I20260929 18:37:56.328974 21765 ts_tablet_manager.cc:1397] T 8c1d8d25b91c456cbf72d6bb0c3869d7 P 1004ea961b344f06b6a5033e2c069359: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260929 18:37:56.328984 21764 log.cc:824] T 8c1d8d25b91c456cbf72d6bb0c3869d7 P 4dc6fe52db3c4e3983ca92bca03b4bbe: Log is configured to *not* fsync() on all Append() calls
I20260929 18:37:56.330585 21764 tablet_bootstrap.cc:492] T 8c1d8d25b91c456cbf72d6bb0c3869d7 P 4dc6fe52db3c4e3983ca92bca03b4bbe: No bootstrap required, opened a new log
I20260929 18:37:56.330772 21764 ts_tablet_manager.cc:1397] T 8c1d8d25b91c456cbf72d6bb0c3869d7 P 4dc6fe52db3c4e3983ca92bca03b4bbe: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260929 18:37:56.332212 21763 raft_consensus.cc:348] T 8c1d8d25b91c456cbf72d6bb0c3869d7 P 9741dfa235754052b0b0c2acc05483e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dc6fe52db3c4e3983ca92bca03b4bbe" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 37145 } } peers { permanent_uuid: "1004ea961b344f06b6a5033e2c069359" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 45113 } } peers { permanent_uuid: "9741dfa235754052b0b0c2acc05483e6" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 44055 } }
I20260929 18:37:56.332487 21763 raft_consensus.cc:374] T 8c1d8d25b91c456cbf72d6bb0c3869d7 P 9741dfa235754052b0b0c2acc05483e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:37:56.332569 21763 raft_consensus.cc:729] T 8c1d8d25b91c456cbf72d6bb0c3869d7 P 9741dfa235754052b0b0c2acc05483e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9741dfa235754052b0b0c2acc05483e6, State: Initialized, Role: FOLLOWER
I20260929 18:37:56.332681 21765 raft_consensus.cc:348] T 8c1d8d25b91c456cbf72d6bb0c3869d7 P 1004ea961b344f06b6a5033e2c069359 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dc6fe52db3c4e3983ca92bca03b4bbe" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 37145 } } peers { permanent_uuid: "1004ea961b344f06b6a5033e2c069359" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 45113 } } peers { permanent_uuid: "9741dfa235754052b0b0c2acc05483e6" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 44055 } }
I20260929 18:37:56.332899 21765 raft_consensus.cc:374] T 8c1d8d25b91c456cbf72d6bb0c3869d7 P 1004ea961b344f06b6a5033e2c069359 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:37:56.333000 21765 raft_consensus.cc:729] T 8c1d8d25b91c456cbf72d6bb0c3869d7 P 1004ea961b344f06b6a5033e2c069359 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1004ea961b344f06b6a5033e2c069359, State: Initialized, Role: FOLLOWER
I20260929 18:37:56.333019 21763 consensus_queue.cc:260] T 8c1d8d25b91c456cbf72d6bb0c3869d7 P 9741dfa235754052b0b0c2acc05483e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dc6fe52db3c4e3983ca92bca03b4bbe" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 37145 } } peers { permanent_uuid: "1004ea961b344f06b6a5033e2c069359" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 45113 } } peers { permanent_uuid: "9741dfa235754052b0b0c2acc05483e6" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 44055 } }
I20260929 18:37:56.333469 21765 consensus_queue.cc:260] T 8c1d8d25b91c456cbf72d6bb0c3869d7 P 1004ea961b344f06b6a5033e2c069359 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dc6fe52db3c4e3983ca92bca03b4bbe" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 37145 } } peers { permanent_uuid: "1004ea961b344f06b6a5033e2c069359" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 45113 } } peers { permanent_uuid: "9741dfa235754052b0b0c2acc05483e6" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 44055 } }
I20260929 18:37:56.334074 21763 ts_tablet_manager.cc:1428] T 8c1d8d25b91c456cbf72d6bb0c3869d7 P 9741dfa235754052b0b0c2acc05483e6: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 18:37:56.334057 21764 raft_consensus.cc:348] T 8c1d8d25b91c456cbf72d6bb0c3869d7 P 4dc6fe52db3c4e3983ca92bca03b4bbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dc6fe52db3c4e3983ca92bca03b4bbe" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 37145 } } peers { permanent_uuid: "1004ea961b344f06b6a5033e2c069359" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 45113 } } peers { permanent_uuid: "9741dfa235754052b0b0c2acc05483e6" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 44055 } }
I20260929 18:37:56.334352 21764 raft_consensus.cc:374] T 8c1d8d25b91c456cbf72d6bb0c3869d7 P 4dc6fe52db3c4e3983ca92bca03b4bbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:37:56.334388 21765 ts_tablet_manager.cc:1428] T 8c1d8d25b91c456cbf72d6bb0c3869d7 P 1004ea961b344f06b6a5033e2c069359: Time spent starting tablet: real 0.005s user 0.000s sys 0.006s
I20260929 18:37:56.334424 21735 heartbeater.cc:503] Master 127.13.100.62:39731 was elected leader, sending a full tablet report...
I20260929 18:37:56.334415 21764 raft_consensus.cc:729] T 8c1d8d25b91c456cbf72d6bb0c3869d7 P 4dc6fe52db3c4e3983ca92bca03b4bbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4dc6fe52db3c4e3983ca92bca03b4bbe, State: Initialized, Role: FOLLOWER
I20260929 18:37:56.334709 21455 heartbeater.cc:503] Master 127.13.100.62:39731 was elected leader, sending a full tablet report...
I20260929 18:37:56.334800 21764 consensus_queue.cc:260] T 8c1d8d25b91c456cbf72d6bb0c3869d7 P 4dc6fe52db3c4e3983ca92bca03b4bbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dc6fe52db3c4e3983ca92bca03b4bbe" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 37145 } } peers { permanent_uuid: "1004ea961b344f06b6a5033e2c069359" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 45113 } } peers { permanent_uuid: "9741dfa235754052b0b0c2acc05483e6" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 44055 } }
I20260929 18:37:56.335394 21595 heartbeater.cc:503] Master 127.13.100.62:39731 was elected leader, sending a full tablet report...
I20260929 18:37:56.335666 21764 ts_tablet_manager.cc:1428] T 8c1d8d25b91c456cbf72d6bb0c3869d7 P 4dc6fe52db3c4e3983ca92bca03b4bbe: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260929 18:37:56.491778 21736 tablet.cc:2369] T 8c1d8d25b91c456cbf72d6bb0c3869d7 P 9741dfa235754052b0b0c2acc05483e6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 18:37:56.528616 21596 tablet.cc:2369] T 8c1d8d25b91c456cbf72d6bb0c3869d7 P 4dc6fe52db3c4e3983ca92bca03b4bbe: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 18:37:56.574443 21456 tablet.cc:2369] T 8c1d8d25b91c456cbf72d6bb0c3869d7 P 1004ea961b344f06b6a5033e2c069359: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:37:56.604650 21770 raft_consensus.cc:482] T 8c1d8d25b91c456cbf72d6bb0c3869d7 P 1004ea961b344f06b6a5033e2c069359 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:37:56.604972 21770 raft_consensus.cc:504] T 8c1d8d25b91c456cbf72d6bb0c3869d7 P 1004ea961b344f06b6a5033e2c069359 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dc6fe52db3c4e3983ca92bca03b4bbe" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 37145 } } peers { permanent_uuid: "1004ea961b344f06b6a5033e2c069359" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 45113 } } peers { permanent_uuid: "9741dfa235754052b0b0c2acc05483e6" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 44055 } }
I20260929 18:37:56.606189 21770 leader_election.cc:288] T 8c1d8d25b91c456cbf72d6bb0c3869d7 P 1004ea961b344f06b6a5033e2c069359 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4dc6fe52db3c4e3983ca92bca03b4bbe (127.13.100.2:37145), 9741dfa235754052b0b0c2acc05483e6 (127.13.100.3:44055)
I20260929 18:37:56.610172 21759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:56.606566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.1:47571 (local address 127.13.100.2:37145)
0929 18:37:56.606738 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:56.606743 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:56.606875 (+ 132us) server_negotiation.cc:407] Connection header received
0929 18:37:56.607062 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:56.607065 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:56.607120 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:56.607214 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:37:56.607491 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:56.608117 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:56.609203 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:56.609621 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:56.609754 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:56.609805 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:56.609983 (+ 178us) server_negotiation.cc:299] Negotiation successful
0929 18:37:56.610046 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 18:37:56.610184 21772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:56.606398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.2:37145 (local address 127.13.100.1:47571)
0929 18:37:56.606681 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:56.606767 (+ 86us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:56.606940 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:56.607238 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:56.607241 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:56.607258 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:37:56.607383 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:56.607388 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:56.608224 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:56.608229 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:56.609075 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:56.609084 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:56.609754 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:56.609757 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:56.609849 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:56.609870 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 18:37:56.609971 (+ 101us) client_negotiation.cc:240] Negotiation successful
0929 18:37:56.610042 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"mutex_wait_us":30,"thread_start_us":98,"threads_started":1}
I20260929 18:37:56.610337 21762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:56.606986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.1:54113 (local address 127.13.100.3:44055)
0929 18:37:56.607107 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:56.607114 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:56.607216 (+ 102us) server_negotiation.cc:407] Connection header received
0929 18:37:56.607364 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:56.607369 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:56.607415 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:56.607499 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:37:56.607749 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:56.608357 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:56.609356 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:56.609743 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:56.609871 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:56.609945 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:56.610127 (+ 182us) server_negotiation.cc:299] Negotiation successful
0929 18:37:56.610190 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 18:37:56.610460 21773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:56.606781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.3:44055 (local address 127.13.100.1:54113)
0929 18:37:56.607102 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:56.607116 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:56.607255 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:56.607512 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:56.607515 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:56.607530 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:37:56.607641 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:56.607646 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:56.608463 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:56.608469 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:56.609232 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:56.609240 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:56.609864 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:56.609867 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:56.609991 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:56.610018 (+ 27us) client_negotiation.cc:769] Sending connection context
0929 18:37:56.610101 (+ 83us) client_negotiation.cc:240] Negotiation successful
0929 18:37:56.610169 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"mutex_wait_us":1,"thread_start_us":73,"threads_started":1}
I20260929 18:37:56.610927 21549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c1d8d25b91c456cbf72d6bb0c3869d7" candidate_uuid: "1004ea961b344f06b6a5033e2c069359" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4dc6fe52db3c4e3983ca92bca03b4bbe" is_pre_election: true
I20260929 18:37:56.611045 21689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c1d8d25b91c456cbf72d6bb0c3869d7" candidate_uuid: "1004ea961b344f06b6a5033e2c069359" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9741dfa235754052b0b0c2acc05483e6" is_pre_election: true
I20260929 18:37:56.611271 21549 raft_consensus.cc:2454] T 8c1d8d25b91c456cbf72d6bb0c3869d7 P 4dc6fe52db3c4e3983ca92bca03b4bbe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1004ea961b344f06b6a5033e2c069359 in term 0.
I20260929 18:37:56.611351 21689 raft_consensus.cc:2454] T 8c1d8d25b91c456cbf72d6bb0c3869d7 P 9741dfa235754052b0b0c2acc05483e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1004ea961b344f06b6a5033e2c069359 in term 0.
I20260929 18:37:56.611734 21342 leader_election.cc:302] T 8c1d8d25b91c456cbf72d6bb0c3869d7 P 1004ea961b344f06b6a5033e2c069359 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1004ea961b344f06b6a5033e2c069359, 4dc6fe52db3c4e3983ca92bca03b4bbe; no voters:
I20260929 18:37:56.612030 21770 raft_consensus.cc:2781] T 8c1d8d25b91c456cbf72d6bb0c3869d7 P 1004ea961b344f06b6a5033e2c069359 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:37:56.612103 21770 raft_consensus.cc:482] T 8c1d8d25b91c456cbf72d6bb0c3869d7 P 1004ea961b344f06b6a5033e2c069359 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:37:56.612133 21770 raft_consensus.cc:3037] T 8c1d8d25b91c456cbf72d6bb0c3869d7 P 1004ea961b344f06b6a5033e2c069359 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:37:56.613137 21770 raft_consensus.cc:504] T 8c1d8d25b91c456cbf72d6bb0c3869d7 P 1004ea961b344f06b6a5033e2c069359 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dc6fe52db3c4e3983ca92bca03b4bbe" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 37145 } } peers { permanent_uuid: "1004ea961b344f06b6a5033e2c069359" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 45113 } } peers { permanent_uuid: "9741dfa235754052b0b0c2acc05483e6" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 44055 } }
I20260929 18:37:56.613476 21770 leader_election.cc:288] T 8c1d8d25b91c456cbf72d6bb0c3869d7 P 1004ea961b344f06b6a5033e2c069359 [CANDIDATE]: Term 1 election: Requested vote from peers 4dc6fe52db3c4e3983ca92bca03b4bbe (127.13.100.2:37145), 9741dfa235754052b0b0c2acc05483e6 (127.13.100.3:44055)
I20260929 18:37:56.613795 21549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c1d8d25b91c456cbf72d6bb0c3869d7" candidate_uuid: "1004ea961b344f06b6a5033e2c069359" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4dc6fe52db3c4e3983ca92bca03b4bbe"
I20260929 18:37:56.613870 21689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c1d8d25b91c456cbf72d6bb0c3869d7" candidate_uuid: "1004ea961b344f06b6a5033e2c069359" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9741dfa235754052b0b0c2acc05483e6"
I20260929 18:37:56.613946 21549 raft_consensus.cc:3037] T 8c1d8d25b91c456cbf72d6bb0c3869d7 P 4dc6fe52db3c4e3983ca92bca03b4bbe [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:37:56.614064 21689 raft_consensus.cc:3037] T 8c1d8d25b91c456cbf72d6bb0c3869d7 P 9741dfa235754052b0b0c2acc05483e6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:37:56.614952 21549 raft_consensus.cc:2454] T 8c1d8d25b91c456cbf72d6bb0c3869d7 P 4dc6fe52db3c4e3983ca92bca03b4bbe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1004ea961b344f06b6a5033e2c069359 in term 1.
I20260929 18:37:56.615232 21342 leader_election.cc:302] T 8c1d8d25b91c456cbf72d6bb0c3869d7 P 1004ea961b344f06b6a5033e2c069359 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1004ea961b344f06b6a5033e2c069359, 4dc6fe52db3c4e3983ca92bca03b4bbe; no voters:
I20260929 18:37:56.615429 21689 raft_consensus.cc:2454] T 8c1d8d25b91c456cbf72d6bb0c3869d7 P 9741dfa235754052b0b0c2acc05483e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1004ea961b344f06b6a5033e2c069359 in term 1.
I20260929 18:37:56.615434 21770 raft_consensus.cc:2781] T 8c1d8d25b91c456cbf72d6bb0c3869d7 P 1004ea961b344f06b6a5033e2c069359 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:37:56.615664 21770 raft_consensus.cc:686] T 8c1d8d25b91c456cbf72d6bb0c3869d7 P 1004ea961b344f06b6a5033e2c069359 [term 1 LEADER]: Becoming Leader. State: Replica: 1004ea961b344f06b6a5033e2c069359, State: Running, Role: LEADER
I20260929 18:37:56.615976 21770 consensus_queue.cc:237] T 8c1d8d25b91c456cbf72d6bb0c3869d7 P 1004ea961b344f06b6a5033e2c069359 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dc6fe52db3c4e3983ca92bca03b4bbe" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 37145 } } peers { permanent_uuid: "1004ea961b344f06b6a5033e2c069359" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 45113 } } peers { permanent_uuid: "9741dfa235754052b0b0c2acc05483e6" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 44055 } }
I20260929 18:37:56.619046 21251 catalog_manager.cc:5495] T 8c1d8d25b91c456cbf72d6bb0c3869d7 P 1004ea961b344f06b6a5033e2c069359 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1004ea961b344f06b6a5033e2c069359 (127.13.100.1). New cstate: current_term: 1 leader_uuid: "1004ea961b344f06b6a5033e2c069359" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dc6fe52db3c4e3983ca92bca03b4bbe" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 37145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1004ea961b344f06b6a5033e2c069359" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 45113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9741dfa235754052b0b0c2acc05483e6" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 44055 } health_report { overall_health: UNKNOWN } } }
I20260929 18:37:56.673710 21761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:56.670098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39410 (local address 127.13.100.1:45113)
0929 18:37:56.670242 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:56.670249 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:56.670267 (+ 18us) server_negotiation.cc:407] Connection header received
0929 18:37:56.670369 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:56.670374 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:56.670444 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:56.670550 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:37:56.670948 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:56.671444 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:56.672325 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:56.672549 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:56.672627 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:56.672888 (+ 261us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:37:56.673300 (+ 412us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:37:56.673382 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:56.673489 (+ 107us) server_negotiation.cc:299] Negotiation successful
0929 18:37:56.673536 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 18:37:56.679247 21689 raft_consensus.cc:1261] T 8c1d8d25b91c456cbf72d6bb0c3869d7 P 9741dfa235754052b0b0c2acc05483e6 [term 1 FOLLOWER]: Refusing update from remote peer 1004ea961b344f06b6a5033e2c069359: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:37:56.679294 21549 raft_consensus.cc:1261] T 8c1d8d25b91c456cbf72d6bb0c3869d7 P 4dc6fe52db3c4e3983ca92bca03b4bbe [term 1 FOLLOWER]: Refusing update from remote peer 1004ea961b344f06b6a5033e2c069359: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:37:56.679929 21770 consensus_queue.cc:1035] T 8c1d8d25b91c456cbf72d6bb0c3869d7 P 1004ea961b344f06b6a5033e2c069359 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9741dfa235754052b0b0c2acc05483e6" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 44055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:37:56.680125 21774 consensus_queue.cc:1035] T 8c1d8d25b91c456cbf72d6bb0c3869d7 P 1004ea961b344f06b6a5033e2c069359 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4dc6fe52db3c4e3983ca92bca03b4bbe" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 37145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:37:56.688402 21777 mvcc.cc:205] Tried to move back new op lower bound from 7334736186071228416 to 7334736185821315072. Current Snapshot: MvccSnapshot[applied={T|T < 7334736186071228416 or (T in {7334736186071228416})}]
I20260929 18:37:56.690093 21778 mvcc.cc:205] Tried to move back new op lower bound from 7334736186071228416 to 7334736185821315072. Current Snapshot: MvccSnapshot[applied={T|T < 7334736186071228416}]
I20260929 18:37:56.694964 21251 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59372:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:37:56.695135 21251 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59372:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:37:56.697458 21251 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 73570d97b6164fdb8636f3f3c01b24e4: Sending DeleteTablet for 3 replicas of tablet 8c1d8d25b91c456cbf72d6bb0c3869d7
I20260929 18:37:56.698201 21529 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c1d8d25b91c456cbf72d6bb0c3869d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:37:56 UTC) from {username='oryx', principal='oryx/127.13.100.62@KRBTEST.COM'} at 127.0.0.1:50074
I20260929 18:37:56.698284 21389 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c1d8d25b91c456cbf72d6bb0c3869d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:37:56 UTC) from {username='oryx', principal='oryx/127.13.100.62@KRBTEST.COM'} at 127.0.0.1:39396
I20260929 18:37:56.698721 21251 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59372:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260929 18:37:56.698885 21788 tablet_replica.cc:330] T 8c1d8d25b91c456cbf72d6bb0c3869d7 P 1004ea961b344f06b6a5033e2c069359: stopping tablet replica
I20260929 18:37:56.698879 21669 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c1d8d25b91c456cbf72d6bb0c3869d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:37:56 UTC) from {username='oryx', principal='oryx/127.13.100.62@KRBTEST.COM'} at 127.0.0.1:48708
W20260929 18:37:56.699142 21251 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:37:56.699167 21788 raft_consensus.cc:2229] T 8c1d8d25b91c456cbf72d6bb0c3869d7 P 1004ea961b344f06b6a5033e2c069359 [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:37:56.699275 21789 tablet_replica.cc:330] T 8c1d8d25b91c456cbf72d6bb0c3869d7 P 9741dfa235754052b0b0c2acc05483e6: stopping tablet replica
I20260929 18:37:56.699484 21788 raft_consensus.cc:2258] T 8c1d8d25b91c456cbf72d6bb0c3869d7 P 1004ea961b344f06b6a5033e2c069359 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:37:56.699555 21789 raft_consensus.cc:2229] T 8c1d8d25b91c456cbf72d6bb0c3869d7 P 9741dfa235754052b0b0c2acc05483e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:37:56.699985 21789 raft_consensus.cc:2258] T 8c1d8d25b91c456cbf72d6bb0c3869d7 P 9741dfa235754052b0b0c2acc05483e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:37:56.701159 21788 ts_tablet_manager.cc:1905] T 8c1d8d25b91c456cbf72d6bb0c3869d7 P 1004ea961b344f06b6a5033e2c069359: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:37:56.701403 21787 tablet_replica.cc:330] T 8c1d8d25b91c456cbf72d6bb0c3869d7 P 4dc6fe52db3c4e3983ca92bca03b4bbe: stopping tablet replica
I20260929 18:37:56.701713 21787 raft_consensus.cc:2229] T 8c1d8d25b91c456cbf72d6bb0c3869d7 P 4dc6fe52db3c4e3983ca92bca03b4bbe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:37:56.702099 21789 ts_tablet_manager.cc:1905] T 8c1d8d25b91c456cbf72d6bb0c3869d7 P 9741dfa235754052b0b0c2acc05483e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:37:56.702646 21787 raft_consensus.cc:2258] T 8c1d8d25b91c456cbf72d6bb0c3869d7 P 4dc6fe52db3c4e3983ca92bca03b4bbe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:37:56.703792 21389 tablet_service.cc:1467] Processing CreateTablet for tablet f21fff2ca04148d19f65024180143c61 (DEFAULT_TABLE table=test-table [id=90cdb37bd9fa4ef1923c6774af0498cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:37:56.704049 21389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f21fff2ca04148d19f65024180143c61. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:37:56.704232 21529 tablet_service.cc:1467] Processing CreateTablet for tablet f21fff2ca04148d19f65024180143c61 (DEFAULT_TABLE table=test-table [id=90cdb37bd9fa4ef1923c6774af0498cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:37:56.704466 21529 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f21fff2ca04148d19f65024180143c61. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:37:56.704861 21789 ts_tablet_manager.cc:1918] T 8c1d8d25b91c456cbf72d6bb0c3869d7 P 9741dfa235754052b0b0c2acc05483e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:37:56.704939 21789 log.cc:1196] T 8c1d8d25b91c456cbf72d6bb0c3869d7 P 9741dfa235754052b0b0c2acc05483e6: Deleting WAL directory at /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707036041078-13712-0/minicluster-data/ts-2/wal/wals/8c1d8d25b91c456cbf72d6bb0c3869d7
I20260929 18:37:56.705251 21789 ts_tablet_manager.cc:1939] T 8c1d8d25b91c456cbf72d6bb0c3869d7 P 9741dfa235754052b0b0c2acc05483e6: Deleting consensus metadata
I20260929 18:37:56.706221 21787 ts_tablet_manager.cc:1905] T 8c1d8d25b91c456cbf72d6bb0c3869d7 P 4dc6fe52db3c4e3983ca92bca03b4bbe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:37:56.706406 21765 tablet_bootstrap.cc:492] T f21fff2ca04148d19f65024180143c61 P 1004ea961b344f06b6a5033e2c069359: Bootstrap starting.
I20260929 18:37:56.706804 21764 tablet_bootstrap.cc:492] T f21fff2ca04148d19f65024180143c61 P 4dc6fe52db3c4e3983ca92bca03b4bbe: Bootstrap starting.
I20260929 18:37:56.707008 21669 tablet_service.cc:1467] Processing CreateTablet for tablet f21fff2ca04148d19f65024180143c61 (DEFAULT_TABLE table=test-table [id=90cdb37bd9fa4ef1923c6774af0498cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:37:56.707268 21669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f21fff2ca04148d19f65024180143c61. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:37:56.707325 21765 tablet_bootstrap.cc:654] T f21fff2ca04148d19f65024180143c61 P 1004ea961b344f06b6a5033e2c069359: Neither blocks nor log segments found. Creating new log.
I20260929 18:37:56.707676 21764 tablet_bootstrap.cc:654] T f21fff2ca04148d19f65024180143c61 P 4dc6fe52db3c4e3983ca92bca03b4bbe: Neither blocks nor log segments found. Creating new log.
I20260929 18:37:56.708523 21765 tablet_bootstrap.cc:492] T f21fff2ca04148d19f65024180143c61 P 1004ea961b344f06b6a5033e2c069359: No bootstrap required, opened a new log
I20260929 18:37:56.708608 21765 ts_tablet_manager.cc:1397] T f21fff2ca04148d19f65024180143c61 P 1004ea961b344f06b6a5033e2c069359: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 18:37:56.708895 21764 tablet_bootstrap.cc:492] T f21fff2ca04148d19f65024180143c61 P 4dc6fe52db3c4e3983ca92bca03b4bbe: No bootstrap required, opened a new log
I20260929 18:37:56.708966 21764 ts_tablet_manager.cc:1397] T f21fff2ca04148d19f65024180143c61 P 4dc6fe52db3c4e3983ca92bca03b4bbe: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 18:37:56.709079 21765 raft_consensus.cc:348] T f21fff2ca04148d19f65024180143c61 P 1004ea961b344f06b6a5033e2c069359 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1004ea961b344f06b6a5033e2c069359" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 45113 } } peers { permanent_uuid: "4dc6fe52db3c4e3983ca92bca03b4bbe" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 37145 } } peers { permanent_uuid: "9741dfa235754052b0b0c2acc05483e6" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 44055 } }
I20260929 18:37:56.709174 21765 raft_consensus.cc:374] T f21fff2ca04148d19f65024180143c61 P 1004ea961b344f06b6a5033e2c069359 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:37:56.709205 21765 raft_consensus.cc:729] T f21fff2ca04148d19f65024180143c61 P 1004ea961b344f06b6a5033e2c069359 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1004ea961b344f06b6a5033e2c069359, State: Initialized, Role: FOLLOWER
I20260929 18:37:56.709326 21765 consensus_queue.cc:260] T f21fff2ca04148d19f65024180143c61 P 1004ea961b344f06b6a5033e2c069359 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1004ea961b344f06b6a5033e2c069359" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 45113 } } peers { permanent_uuid: "4dc6fe52db3c4e3983ca92bca03b4bbe" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 37145 } } peers { permanent_uuid: "9741dfa235754052b0b0c2acc05483e6" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 44055 } }
I20260929 18:37:56.709388 21764 raft_consensus.cc:348] T f21fff2ca04148d19f65024180143c61 P 4dc6fe52db3c4e3983ca92bca03b4bbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1004ea961b344f06b6a5033e2c069359" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 45113 } } peers { permanent_uuid: "4dc6fe52db3c4e3983ca92bca03b4bbe" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 37145 } } peers { permanent_uuid: "9741dfa235754052b0b0c2acc05483e6" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 44055 } }
I20260929 18:37:56.709488 21764 raft_consensus.cc:374] T f21fff2ca04148d19f65024180143c61 P 4dc6fe52db3c4e3983ca92bca03b4bbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:37:56.709520 21764 raft_consensus.cc:729] T f21fff2ca04148d19f65024180143c61 P 4dc6fe52db3c4e3983ca92bca03b4bbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4dc6fe52db3c4e3983ca92bca03b4bbe, State: Initialized, Role: FOLLOWER
I20260929 18:37:56.709609 21765 ts_tablet_manager.cc:1428] T f21fff2ca04148d19f65024180143c61 P 1004ea961b344f06b6a5033e2c069359: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 18:37:56.709658 21763 tablet_bootstrap.cc:492] T f21fff2ca04148d19f65024180143c61 P 9741dfa235754052b0b0c2acc05483e6: Bootstrap starting.
I20260929 18:37:56.709640 21764 consensus_queue.cc:260] T f21fff2ca04148d19f65024180143c61 P 4dc6fe52db3c4e3983ca92bca03b4bbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1004ea961b344f06b6a5033e2c069359" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 45113 } } peers { permanent_uuid: "4dc6fe52db3c4e3983ca92bca03b4bbe" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 37145 } } peers { permanent_uuid: "9741dfa235754052b0b0c2acc05483e6" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 44055 } }
I20260929 18:37:56.709872 21764 ts_tablet_manager.cc:1428] T f21fff2ca04148d19f65024180143c61 P 4dc6fe52db3c4e3983ca92bca03b4bbe: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 18:37:56.709983 21788 ts_tablet_manager.cc:1918] T 8c1d8d25b91c456cbf72d6bb0c3869d7 P 1004ea961b344f06b6a5033e2c069359: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:37:56.710068 21788 log.cc:1196] T 8c1d8d25b91c456cbf72d6bb0c3869d7 P 1004ea961b344f06b6a5033e2c069359: Deleting WAL directory at /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707036041078-13712-0/minicluster-data/ts-0/wal/wals/8c1d8d25b91c456cbf72d6bb0c3869d7
I20260929 18:37:56.710372 21788 ts_tablet_manager.cc:1939] T 8c1d8d25b91c456cbf72d6bb0c3869d7 P 1004ea961b344f06b6a5033e2c069359: Deleting consensus metadata
I20260929 18:37:56.710702 21238 catalog_manager.cc:4841] TS 9741dfa235754052b0b0c2acc05483e6 (127.13.100.3:44055): tablet 8c1d8d25b91c456cbf72d6bb0c3869d7 (table test-table [id=3325c13033f74e66b10b375be4e5e09d]) successfully deleted
I20260929 18:37:56.711544 21237 catalog_manager.cc:4841] TS 1004ea961b344f06b6a5033e2c069359 (127.13.100.1:45113): tablet 8c1d8d25b91c456cbf72d6bb0c3869d7 (table test-table [id=3325c13033f74e66b10b375be4e5e09d]) successfully deleted
I20260929 18:37:56.711830 21763 tablet_bootstrap.cc:654] T f21fff2ca04148d19f65024180143c61 P 9741dfa235754052b0b0c2acc05483e6: Neither blocks nor log segments found. Creating new log.
I20260929 18:37:56.712157 21787 ts_tablet_manager.cc:1918] T 8c1d8d25b91c456cbf72d6bb0c3869d7 P 4dc6fe52db3c4e3983ca92bca03b4bbe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:37:56.712255 21787 log.cc:1196] T 8c1d8d25b91c456cbf72d6bb0c3869d7 P 4dc6fe52db3c4e3983ca92bca03b4bbe: Deleting WAL directory at /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707036041078-13712-0/minicluster-data/ts-1/wal/wals/8c1d8d25b91c456cbf72d6bb0c3869d7
I20260929 18:37:56.712592 21787 ts_tablet_manager.cc:1939] T 8c1d8d25b91c456cbf72d6bb0c3869d7 P 4dc6fe52db3c4e3983ca92bca03b4bbe: Deleting consensus metadata
I20260929 18:37:56.713115 21763 tablet_bootstrap.cc:492] T f21fff2ca04148d19f65024180143c61 P 9741dfa235754052b0b0c2acc05483e6: No bootstrap required, opened a new log
I20260929 18:37:56.713203 21763 ts_tablet_manager.cc:1397] T f21fff2ca04148d19f65024180143c61 P 9741dfa235754052b0b0c2acc05483e6: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260929 18:37:56.713727 21763 raft_consensus.cc:348] T f21fff2ca04148d19f65024180143c61 P 9741dfa235754052b0b0c2acc05483e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1004ea961b344f06b6a5033e2c069359" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 45113 } } peers { permanent_uuid: "4dc6fe52db3c4e3983ca92bca03b4bbe" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 37145 } } peers { permanent_uuid: "9741dfa235754052b0b0c2acc05483e6" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 44055 } }
I20260929 18:37:56.713850 21763 raft_consensus.cc:374] T f21fff2ca04148d19f65024180143c61 P 9741dfa235754052b0b0c2acc05483e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:37:56.713909 21763 raft_consensus.cc:729] T f21fff2ca04148d19f65024180143c61 P 9741dfa235754052b0b0c2acc05483e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9741dfa235754052b0b0c2acc05483e6, State: Initialized, Role: FOLLOWER
I20260929 18:37:56.714011 21238 catalog_manager.cc:4841] TS 4dc6fe52db3c4e3983ca92bca03b4bbe (127.13.100.2:37145): tablet 8c1d8d25b91c456cbf72d6bb0c3869d7 (table test-table [id=3325c13033f74e66b10b375be4e5e09d]) successfully deleted
I20260929 18:37:56.714046 21763 consensus_queue.cc:260] T f21fff2ca04148d19f65024180143c61 P 9741dfa235754052b0b0c2acc05483e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1004ea961b344f06b6a5033e2c069359" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 45113 } } peers { permanent_uuid: "4dc6fe52db3c4e3983ca92bca03b4bbe" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 37145 } } peers { permanent_uuid: "9741dfa235754052b0b0c2acc05483e6" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 44055 } }
I20260929 18:37:56.714291 21763 ts_tablet_manager.cc:1428] T f21fff2ca04148d19f65024180143c61 P 9741dfa235754052b0b0c2acc05483e6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 18:37:56.869688 21769 raft_consensus.cc:482] T f21fff2ca04148d19f65024180143c61 P 9741dfa235754052b0b0c2acc05483e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:37:56.869841 21769 raft_consensus.cc:504] T f21fff2ca04148d19f65024180143c61 P 9741dfa235754052b0b0c2acc05483e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1004ea961b344f06b6a5033e2c069359" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 45113 } } peers { permanent_uuid: "4dc6fe52db3c4e3983ca92bca03b4bbe" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 37145 } } peers { permanent_uuid: "9741dfa235754052b0b0c2acc05483e6" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 44055 } }
I20260929 18:37:56.870961 21769 leader_election.cc:288] T f21fff2ca04148d19f65024180143c61 P 9741dfa235754052b0b0c2acc05483e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1004ea961b344f06b6a5033e2c069359 (127.13.100.1:45113), 4dc6fe52db3c4e3983ca92bca03b4bbe (127.13.100.2:37145)
I20260929 18:37:56.876394 21759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:56.871363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.3:33199 (local address 127.13.100.2:37145)
0929 18:37:56.871512 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:56.871518 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:56.871693 (+ 175us) server_negotiation.cc:407] Connection header received
0929 18:37:56.871753 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:56.871755 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:56.871804 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:56.871895 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:37:56.872317 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:56.873115 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:56.875073 (+ 1958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:56.875697 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:56.875825 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:56.875925 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:56.876195 (+ 270us) server_negotiation.cc:299] Negotiation successful
0929 18:37:56.876265 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 18:37:56.876394 21761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:56.871305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.3:60571 (local address 127.13.100.1:45113)
0929 18:37:56.871513 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:56.871519 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:56.871580 (+ 61us) server_negotiation.cc:407] Connection header received
0929 18:37:56.871749 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:56.871752 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:56.871804 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:56.871890 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:37:56.872317 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:56.873099 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:56.875073 (+ 1974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:56.875697 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:56.875825 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:56.875879 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:56.876195 (+ 316us) server_negotiation.cc:299] Negotiation successful
0929 18:37:56.876265 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119}
I20260929 18:37:56.876456 21793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:56.871222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.1:45113 (local address 127.13.100.3:60571)
0929 18:37:56.871494 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:56.871509 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:56.871630 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:56.871919 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:56.871921 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:56.871959 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:37:56.872175 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:56.872181 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:56.873255 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:56.873259 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:56.874906 (+ 1647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:56.874917 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:56.875846 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:56.875848 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:56.876027 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:56.876076 (+ 49us) client_negotiation.cc:769] Sending connection context
0929 18:37:56.876160 (+ 84us) client_negotiation.cc:240] Negotiation successful
0929 18:37:56.876265 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"mutex_wait_us":336,"spinlock_wait_cycles":25600,"thread_start_us":91,"threads_started":1}
I20260929 18:37:56.876458 21794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:56.871264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.2:37145 (local address 127.13.100.3:33199)
0929 18:37:56.871593 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:56.871605 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:56.871683 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:56.871918 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:56.871921 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:56.871961 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:37:56.872167 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:56.872176 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:56.873255 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:56.873259 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:56.874910 (+ 1651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:56.874917 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:56.875842 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:56.875846 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:56.876000 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:56.876076 (+ 76us) client_negotiation.cc:769] Sending connection context
0929 18:37:56.876166 (+ 90us) client_negotiation.cc:240] Negotiation successful
0929 18:37:56.876265 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"mutex_wait_us":321,"spinlock_wait_cycles":4608,"thread_start_us":88,"threads_started":1}
I20260929 18:37:56.876875 21549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f21fff2ca04148d19f65024180143c61" candidate_uuid: "9741dfa235754052b0b0c2acc05483e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4dc6fe52db3c4e3983ca92bca03b4bbe" is_pre_election: true
I20260929 18:37:56.877055 21549 raft_consensus.cc:2454] T f21fff2ca04148d19f65024180143c61 P 4dc6fe52db3c4e3983ca92bca03b4bbe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9741dfa235754052b0b0c2acc05483e6 in term 0.
I20260929 18:37:56.877084 21409 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f21fff2ca04148d19f65024180143c61" candidate_uuid: "9741dfa235754052b0b0c2acc05483e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1004ea961b344f06b6a5033e2c069359" is_pre_election: true
I20260929 18:37:56.877328 21409 raft_consensus.cc:2454] T f21fff2ca04148d19f65024180143c61 P 1004ea961b344f06b6a5033e2c069359 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9741dfa235754052b0b0c2acc05483e6 in term 0.
I20260929 18:37:56.877471 21622 leader_election.cc:302] T f21fff2ca04148d19f65024180143c61 P 9741dfa235754052b0b0c2acc05483e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4dc6fe52db3c4e3983ca92bca03b4bbe, 9741dfa235754052b0b0c2acc05483e6; no voters:
I20260929 18:37:56.877751 21769 raft_consensus.cc:2781] T f21fff2ca04148d19f65024180143c61 P 9741dfa235754052b0b0c2acc05483e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:37:56.877838 21769 raft_consensus.cc:482] T f21fff2ca04148d19f65024180143c61 P 9741dfa235754052b0b0c2acc05483e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:37:56.877923 21769 raft_consensus.cc:3037] T f21fff2ca04148d19f65024180143c61 P 9741dfa235754052b0b0c2acc05483e6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:37:56.878862 21769 raft_consensus.cc:504] T f21fff2ca04148d19f65024180143c61 P 9741dfa235754052b0b0c2acc05483e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1004ea961b344f06b6a5033e2c069359" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 45113 } } peers { permanent_uuid: "4dc6fe52db3c4e3983ca92bca03b4bbe" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 37145 } } peers { permanent_uuid: "9741dfa235754052b0b0c2acc05483e6" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 44055 } }
I20260929 18:37:56.879195 21769 leader_election.cc:288] T f21fff2ca04148d19f65024180143c61 P 9741dfa235754052b0b0c2acc05483e6 [CANDIDATE]: Term 1 election: Requested vote from peers 1004ea961b344f06b6a5033e2c069359 (127.13.100.1:45113), 4dc6fe52db3c4e3983ca92bca03b4bbe (127.13.100.2:37145)
I20260929 18:37:56.879529 21409 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f21fff2ca04148d19f65024180143c61" candidate_uuid: "9741dfa235754052b0b0c2acc05483e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1004ea961b344f06b6a5033e2c069359"
I20260929 18:37:56.879575 21549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f21fff2ca04148d19f65024180143c61" candidate_uuid: "9741dfa235754052b0b0c2acc05483e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4dc6fe52db3c4e3983ca92bca03b4bbe"
I20260929 18:37:56.879653 21409 raft_consensus.cc:3037] T f21fff2ca04148d19f65024180143c61 P 1004ea961b344f06b6a5033e2c069359 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:37:56.879680 21549 raft_consensus.cc:3037] T f21fff2ca04148d19f65024180143c61 P 4dc6fe52db3c4e3983ca92bca03b4bbe [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:37:56.880705 21409 raft_consensus.cc:2454] T f21fff2ca04148d19f65024180143c61 P 1004ea961b344f06b6a5033e2c069359 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9741dfa235754052b0b0c2acc05483e6 in term 1.
I20260929 18:37:56.880705 21549 raft_consensus.cc:2454] T f21fff2ca04148d19f65024180143c61 P 4dc6fe52db3c4e3983ca92bca03b4bbe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9741dfa235754052b0b0c2acc05483e6 in term 1.
I20260929 18:37:56.881026 21621 leader_election.cc:302] T f21fff2ca04148d19f65024180143c61 P 9741dfa235754052b0b0c2acc05483e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1004ea961b344f06b6a5033e2c069359, 9741dfa235754052b0b0c2acc05483e6; no voters:
I20260929 18:37:56.881222 21769 raft_consensus.cc:2781] T f21fff2ca04148d19f65024180143c61 P 9741dfa235754052b0b0c2acc05483e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:37:56.881371 21769 raft_consensus.cc:686] T f21fff2ca04148d19f65024180143c61 P 9741dfa235754052b0b0c2acc05483e6 [term 1 LEADER]: Becoming Leader. State: Replica: 9741dfa235754052b0b0c2acc05483e6, State: Running, Role: LEADER
I20260929 18:37:56.881660 21769 consensus_queue.cc:237] T f21fff2ca04148d19f65024180143c61 P 9741dfa235754052b0b0c2acc05483e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1004ea961b344f06b6a5033e2c069359" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 45113 } } peers { permanent_uuid: "4dc6fe52db3c4e3983ca92bca03b4bbe" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 37145 } } peers { permanent_uuid: "9741dfa235754052b0b0c2acc05483e6" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 44055 } }
I20260929 18:37:56.883919 21252 catalog_manager.cc:5495] T f21fff2ca04148d19f65024180143c61 P 9741dfa235754052b0b0c2acc05483e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9741dfa235754052b0b0c2acc05483e6 (127.13.100.3). New cstate: current_term: 1 leader_uuid: "9741dfa235754052b0b0c2acc05483e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1004ea961b344f06b6a5033e2c069359" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 45113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4dc6fe52db3c4e3983ca92bca03b4bbe" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 37145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9741dfa235754052b0b0c2acc05483e6" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 44055 } health_report { overall_health: HEALTHY } } }
Sep 29 18:37:56 dist-test-slave-j1xs krb5kdc[21207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707075, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.100.62@KRBTEST.COM for oryx/127.13.100.62@KRBTEST.COM
I20260929 18:37:56.931211 21802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:56.923502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:39731 (local address 127.0.0.1:59376)
0929 18:37:56.923779 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:56.923791 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:56.923876 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:56.924097 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:56.924099 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:56.924448 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:37:56.924540 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:56.924545 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:56.925287 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:56.925290 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:56.925825 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:56.925832 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:56.926329 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:56.926333 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:56.926409 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:56.926914 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:37:56.926925 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:37:56.928346 (+ 1421us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:37:56.929816 (+ 1470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:56.929823 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:56.929828 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:56.930105 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:56.930338 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:56.930340 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:56.930341 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:56.930414 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:56.930727 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:37:56.930733 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:37:56.930821 (+ 88us) client_negotiation.cc:769] Sending connection context
0929 18:37:56.930906 (+ 85us) client_negotiation.cc:240] Negotiation successful
0929 18:37:56.931022 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":119,"threads_started":1}
I20260929 18:37:56.931213 21803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:56.923538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59376 (local address 127.13.100.62:39731)
0929 18:37:56.923878 (+ 340us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:56.923884 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:56.923900 (+ 16us) server_negotiation.cc:407] Connection header received
0929 18:37:56.923954 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:56.923956 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:56.923994 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:56.924061 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:56.924664 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:56.925190 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:56.925970 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:56.926195 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:56.926273 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:56.928470 (+ 2197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:56.928473 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:56.928475 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:37:56.928494 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:56.929724 (+ 1230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:56.930187 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:56.930189 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:56.930191 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:56.930260 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:56.930488 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:56.930490 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:56.930492 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:56.930640 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:56.930712 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:56.930935 (+ 223us) server_negotiation.cc:299] Negotiation successful
0929 18:37:56.931075 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":29,"server-negotiator.queue_time_us":286,"thread_start_us":93,"threads_started":1}
I20260929 18:37:56.934974 21252 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.13.100.62@KRBTEST.COM'} at 127.0.0.1:59376:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 18:37:56.935596 21252 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:37:56.940054 21389 tablet_service.cc:1467] Processing CreateTablet for tablet b9d45300961942b281d8403e89e8a4f8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8b6a70297e8f4a0fa800c6ed9987b6d8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 18:37:56.940156 21529 tablet_service.cc:1467] Processing CreateTablet for tablet b9d45300961942b281d8403e89e8a4f8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8b6a70297e8f4a0fa800c6ed9987b6d8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 18:37:56.940290 21669 tablet_service.cc:1467] Processing CreateTablet for tablet b9d45300961942b281d8403e89e8a4f8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8b6a70297e8f4a0fa800c6ed9987b6d8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 18:37:56.940428 21389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9d45300961942b281d8403e89e8a4f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:37:56.940539 21529 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9d45300961942b281d8403e89e8a4f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:37:56.940585 21669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9d45300961942b281d8403e89e8a4f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:37:56.942871 21765 tablet_bootstrap.cc:492] T b9d45300961942b281d8403e89e8a4f8 P 1004ea961b344f06b6a5033e2c069359: Bootstrap starting.
I20260929 18:37:56.943212 21763 tablet_bootstrap.cc:492] T b9d45300961942b281d8403e89e8a4f8 P 9741dfa235754052b0b0c2acc05483e6: Bootstrap starting.
I20260929 18:37:56.943248 21764 tablet_bootstrap.cc:492] T b9d45300961942b281d8403e89e8a4f8 P 4dc6fe52db3c4e3983ca92bca03b4bbe: Bootstrap starting.
I20260929 18:37:56.943727 21765 tablet_bootstrap.cc:654] T b9d45300961942b281d8403e89e8a4f8 P 1004ea961b344f06b6a5033e2c069359: Neither blocks nor log segments found. Creating new log.
I20260929 18:37:56.944126 21763 tablet_bootstrap.cc:654] T b9d45300961942b281d8403e89e8a4f8 P 9741dfa235754052b0b0c2acc05483e6: Neither blocks nor log segments found. Creating new log.
I20260929 18:37:56.944206 21764 tablet_bootstrap.cc:654] T b9d45300961942b281d8403e89e8a4f8 P 4dc6fe52db3c4e3983ca92bca03b4bbe: Neither blocks nor log segments found. Creating new log.
I20260929 18:37:56.945099 21765 tablet_bootstrap.cc:492] T b9d45300961942b281d8403e89e8a4f8 P 1004ea961b344f06b6a5033e2c069359: No bootstrap required, opened a new log
I20260929 18:37:56.945188 21765 ts_tablet_manager.cc:1397] T b9d45300961942b281d8403e89e8a4f8 P 1004ea961b344f06b6a5033e2c069359: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260929 18:37:56.945299 21764 tablet_bootstrap.cc:492] T b9d45300961942b281d8403e89e8a4f8 P 4dc6fe52db3c4e3983ca92bca03b4bbe: No bootstrap required, opened a new log
I20260929 18:37:56.945374 21764 ts_tablet_manager.cc:1397] T b9d45300961942b281d8403e89e8a4f8 P 4dc6fe52db3c4e3983ca92bca03b4bbe: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260929 18:37:56.945390 21763 tablet_bootstrap.cc:492] T b9d45300961942b281d8403e89e8a4f8 P 9741dfa235754052b0b0c2acc05483e6: No bootstrap required, opened a new log
I20260929 18:37:56.945472 21763 ts_tablet_manager.cc:1397] T b9d45300961942b281d8403e89e8a4f8 P 9741dfa235754052b0b0c2acc05483e6: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 18:37:56.945598 21765 raft_consensus.cc:348] T b9d45300961942b281d8403e89e8a4f8 P 1004ea961b344f06b6a5033e2c069359 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1004ea961b344f06b6a5033e2c069359" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 45113 } } peers { permanent_uuid: "4dc6fe52db3c4e3983ca92bca03b4bbe" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 37145 } } peers { permanent_uuid: "9741dfa235754052b0b0c2acc05483e6" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 44055 } }
I20260929 18:37:56.945715 21765 raft_consensus.cc:374] T b9d45300961942b281d8403e89e8a4f8 P 1004ea961b344f06b6a5033e2c069359 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:37:56.945757 21765 raft_consensus.cc:729] T b9d45300961942b281d8403e89e8a4f8 P 1004ea961b344f06b6a5033e2c069359 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1004ea961b344f06b6a5033e2c069359, State: Initialized, Role: FOLLOWER
I20260929 18:37:56.945782 21764 raft_consensus.cc:348] T b9d45300961942b281d8403e89e8a4f8 P 4dc6fe52db3c4e3983ca92bca03b4bbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1004ea961b344f06b6a5033e2c069359" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 45113 } } peers { permanent_uuid: "4dc6fe52db3c4e3983ca92bca03b4bbe" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 37145 } } peers { permanent_uuid: "9741dfa235754052b0b0c2acc05483e6" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 44055 } }
I20260929 18:37:56.945879 21764 raft_consensus.cc:374] T b9d45300961942b281d8403e89e8a4f8 P 4dc6fe52db3c4e3983ca92bca03b4bbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:37:56.945947 21764 raft_consensus.cc:729] T b9d45300961942b281d8403e89e8a4f8 P 4dc6fe52db3c4e3983ca92bca03b4bbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4dc6fe52db3c4e3983ca92bca03b4bbe, State: Initialized, Role: FOLLOWER
I20260929 18:37:56.945914 21765 consensus_queue.cc:260] T b9d45300961942b281d8403e89e8a4f8 P 1004ea961b344f06b6a5033e2c069359 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1004ea961b344f06b6a5033e2c069359" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 45113 } } peers { permanent_uuid: "4dc6fe52db3c4e3983ca92bca03b4bbe" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 37145 } } peers { permanent_uuid: "9741dfa235754052b0b0c2acc05483e6" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 44055 } }
I20260929 18:37:56.945957 21763 raft_consensus.cc:348] T b9d45300961942b281d8403e89e8a4f8 P 9741dfa235754052b0b0c2acc05483e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1004ea961b344f06b6a5033e2c069359" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 45113 } } peers { permanent_uuid: "4dc6fe52db3c4e3983ca92bca03b4bbe" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 37145 } } peers { permanent_uuid: "9741dfa235754052b0b0c2acc05483e6" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 44055 } }
I20260929 18:37:56.946072 21763 raft_consensus.cc:374] T b9d45300961942b281d8403e89e8a4f8 P 9741dfa235754052b0b0c2acc05483e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:37:56.946106 21763 raft_consensus.cc:729] T b9d45300961942b281d8403e89e8a4f8 P 9741dfa235754052b0b0c2acc05483e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9741dfa235754052b0b0c2acc05483e6, State: Initialized, Role: FOLLOWER
I20260929 18:37:56.946069 21764 consensus_queue.cc:260] T b9d45300961942b281d8403e89e8a4f8 P 4dc6fe52db3c4e3983ca92bca03b4bbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1004ea961b344f06b6a5033e2c069359" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 45113 } } peers { permanent_uuid: "4dc6fe52db3c4e3983ca92bca03b4bbe" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 37145 } } peers { permanent_uuid: "9741dfa235754052b0b0c2acc05483e6" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 44055 } }
I20260929 18:37:56.946187 21774 tablet_replica.cc:439] T b9d45300961942b281d8403e89e8a4f8 P 1004ea961b344f06b6a5033e2c069359: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1004ea961b344f06b6a5033e2c069359" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 45113 } } peers { permanent_uuid: "4dc6fe52db3c4e3983ca92bca03b4bbe" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 37145 } } peers { permanent_uuid: "9741dfa235754052b0b0c2acc05483e6" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 44055 } } }
I20260929 18:37:56.946221 21763 consensus_queue.cc:260] T b9d45300961942b281d8403e89e8a4f8 P 9741dfa235754052b0b0c2acc05483e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1004ea961b344f06b6a5033e2c069359" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 45113 } } peers { permanent_uuid: "4dc6fe52db3c4e3983ca92bca03b4bbe" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 37145 } } peers { permanent_uuid: "9741dfa235754052b0b0c2acc05483e6" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 44055 } }
I20260929 18:37:56.946318 21774 tablet_replica.cc:442] T b9d45300961942b281d8403e89e8a4f8 P 1004ea961b344f06b6a5033e2c069359: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 18:37:56.946321 21764 ts_tablet_manager.cc:1428] T b9d45300961942b281d8403e89e8a4f8 P 4dc6fe52db3c4e3983ca92bca03b4bbe: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260929 18:37:56.946478 21763 ts_tablet_manager.cc:1428] T b9d45300961942b281d8403e89e8a4f8 P 9741dfa235754052b0b0c2acc05483e6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 18:37:56.946525 21771 tablet_replica.cc:439] T b9d45300961942b281d8403e89e8a4f8 P 4dc6fe52db3c4e3983ca92bca03b4bbe: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1004ea961b344f06b6a5033e2c069359" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 45113 } } peers { permanent_uuid: "4dc6fe52db3c4e3983ca92bca03b4bbe" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 37145 } } peers { permanent_uuid: "9741dfa235754052b0b0c2acc05483e6" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 44055 } } }
I20260929 18:37:56.946631 21771 tablet_replica.cc:442] T b9d45300961942b281d8403e89e8a4f8 P 4dc6fe52db3c4e3983ca92bca03b4bbe: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 18:37:56.946659 21769 tablet_replica.cc:439] T b9d45300961942b281d8403e89e8a4f8 P 9741dfa235754052b0b0c2acc05483e6: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1004ea961b344f06b6a5033e2c069359" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 45113 } } peers { permanent_uuid: "4dc6fe52db3c4e3983ca92bca03b4bbe" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 37145 } } peers { permanent_uuid: "9741dfa235754052b0b0c2acc05483e6" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 44055 } } }
I20260929 18:37:56.946768 21769 tablet_replica.cc:442] T b9d45300961942b281d8403e89e8a4f8 P 9741dfa235754052b0b0c2acc05483e6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 18:37:56.946199 21765 ts_tablet_manager.cc:1428] T b9d45300961942b281d8403e89e8a4f8 P 1004ea961b344f06b6a5033e2c069359: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 18:37:57.044678 21771 raft_consensus.cc:482] T b9d45300961942b281d8403e89e8a4f8 P 4dc6fe52db3c4e3983ca92bca03b4bbe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:37:57.044816 21771 raft_consensus.cc:504] T b9d45300961942b281d8403e89e8a4f8 P 4dc6fe52db3c4e3983ca92bca03b4bbe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1004ea961b344f06b6a5033e2c069359" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 45113 } } peers { permanent_uuid: "4dc6fe52db3c4e3983ca92bca03b4bbe" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 37145 } } peers { permanent_uuid: "9741dfa235754052b0b0c2acc05483e6" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 44055 } }
I20260929 18:37:57.045960 21771 leader_election.cc:288] T b9d45300961942b281d8403e89e8a4f8 P 4dc6fe52db3c4e3983ca92bca03b4bbe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1004ea961b344f06b6a5033e2c069359 (127.13.100.1:45113), 9741dfa235754052b0b0c2acc05483e6 (127.13.100.3:44055)
I20260929 18:37:57.050074 21808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:57.046336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.3:44055 (local address 127.13.100.2:40493)
0929 18:37:57.046748 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:57.046760 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:57.046903 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:57.047191 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:57.047193 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:57.047209 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:37:57.047313 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:57.047326 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:57.048052 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:57.048056 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:57.048918 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:57.048926 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:57.049583 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:57.049587 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:57.049683 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:57.049706 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 18:37:57.049802 (+ 96us) client_negotiation.cc:240] Negotiation successful
0929 18:37:57.049893 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"mutex_wait_us":29,"thread_start_us":138,"threads_started":1}
I20260929 18:37:57.050053 21762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:57.046439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.2:40493 (local address 127.13.100.3:44055)
0929 18:37:57.046569 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:57.046575 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:57.046857 (+ 282us) server_negotiation.cc:407] Connection header received
0929 18:37:57.047028 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:57.047033 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:57.047084 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:57.047177 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:37:57.047432 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:57.047925 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:57.049058 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:57.049442 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:57.049590 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:57.049650 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:57.049829 (+ 179us) server_negotiation.cc:299] Negotiation successful
0929 18:37:57.049928 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 18:37:57.050328 21761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:57.046532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.2:44171 (local address 127.13.100.1:45113)
0929 18:37:57.046644 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:57.046650 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:57.047027 (+ 377us) server_negotiation.cc:407] Connection header received
0929 18:37:57.047191 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:57.047195 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:57.047248 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:57.047347 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:37:57.047608 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:57.048121 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:57.049222 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:57.049653 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:57.049793 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:57.049850 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:57.050078 (+ 228us) server_negotiation.cc:299] Negotiation successful
0929 18:37:57.050193 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 18:37:57.050390 21807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:57.046335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.1:45113 (local address 127.13.100.2:44171)
0929 18:37:57.046906 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:57.046919 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:57.047061 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:57.047357 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:57.047360 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:57.047377 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:37:57.047486 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:57.047492 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:57.048270 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:57.048274 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:57.049082 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:57.049091 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:57.049793 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:57.049797 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:57.049949 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:57.049968 (+ 19us) client_negotiation.cc:769] Sending connection context
0929 18:37:57.050064 (+ 96us) client_negotiation.cc:240] Negotiation successful
0929 18:37:57.050183 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"mutex_wait_us":84,"thread_start_us":94,"threads_started":1}
I20260929 18:37:57.050932 21409 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9d45300961942b281d8403e89e8a4f8" candidate_uuid: "4dc6fe52db3c4e3983ca92bca03b4bbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1004ea961b344f06b6a5033e2c069359" is_pre_election: true
I20260929 18:37:57.051100 21409 raft_consensus.cc:2454] T b9d45300961942b281d8403e89e8a4f8 P 1004ea961b344f06b6a5033e2c069359 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4dc6fe52db3c4e3983ca92bca03b4bbe in term 0.
I20260929 18:37:57.051189 21689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9d45300961942b281d8403e89e8a4f8" candidate_uuid: "4dc6fe52db3c4e3983ca92bca03b4bbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9741dfa235754052b0b0c2acc05483e6" is_pre_election: true
I20260929 18:37:57.051319 21689 raft_consensus.cc:2454] T b9d45300961942b281d8403e89e8a4f8 P 9741dfa235754052b0b0c2acc05483e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4dc6fe52db3c4e3983ca92bca03b4bbe in term 0.
I20260929 18:37:57.051573 21481 leader_election.cc:302] T b9d45300961942b281d8403e89e8a4f8 P 4dc6fe52db3c4e3983ca92bca03b4bbe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1004ea961b344f06b6a5033e2c069359, 4dc6fe52db3c4e3983ca92bca03b4bbe; no voters:
I20260929 18:37:57.051857 21771 raft_consensus.cc:2781] T b9d45300961942b281d8403e89e8a4f8 P 4dc6fe52db3c4e3983ca92bca03b4bbe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:37:57.051916 21771 raft_consensus.cc:482] T b9d45300961942b281d8403e89e8a4f8 P 4dc6fe52db3c4e3983ca92bca03b4bbe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:37:57.051954 21771 raft_consensus.cc:3037] T b9d45300961942b281d8403e89e8a4f8 P 4dc6fe52db3c4e3983ca92bca03b4bbe [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:37:57.053306 21771 raft_consensus.cc:504] T b9d45300961942b281d8403e89e8a4f8 P 4dc6fe52db3c4e3983ca92bca03b4bbe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1004ea961b344f06b6a5033e2c069359" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 45113 } } peers { permanent_uuid: "4dc6fe52db3c4e3983ca92bca03b4bbe" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 37145 } } peers { permanent_uuid: "9741dfa235754052b0b0c2acc05483e6" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 44055 } }
I20260929 18:37:57.053668 21771 leader_election.cc:288] T b9d45300961942b281d8403e89e8a4f8 P 4dc6fe52db3c4e3983ca92bca03b4bbe [CANDIDATE]: Term 1 election: Requested vote from peers 1004ea961b344f06b6a5033e2c069359 (127.13.100.1:45113), 9741dfa235754052b0b0c2acc05483e6 (127.13.100.3:44055)
I20260929 18:37:57.054059 21409 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9d45300961942b281d8403e89e8a4f8" candidate_uuid: "4dc6fe52db3c4e3983ca92bca03b4bbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1004ea961b344f06b6a5033e2c069359"
I20260929 18:37:57.054207 21409 raft_consensus.cc:3037] T b9d45300961942b281d8403e89e8a4f8 P 1004ea961b344f06b6a5033e2c069359 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:37:57.054177 21689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9d45300961942b281d8403e89e8a4f8" candidate_uuid: "4dc6fe52db3c4e3983ca92bca03b4bbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9741dfa235754052b0b0c2acc05483e6"
I20260929 18:37:57.054364 21689 raft_consensus.cc:3037] T b9d45300961942b281d8403e89e8a4f8 P 9741dfa235754052b0b0c2acc05483e6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:37:57.055142 21409 raft_consensus.cc:2454] T b9d45300961942b281d8403e89e8a4f8 P 1004ea961b344f06b6a5033e2c069359 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4dc6fe52db3c4e3983ca92bca03b4bbe in term 1.
I20260929 18:37:57.055420 21689 raft_consensus.cc:2454] T b9d45300961942b281d8403e89e8a4f8 P 9741dfa235754052b0b0c2acc05483e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4dc6fe52db3c4e3983ca92bca03b4bbe in term 1.
I20260929 18:37:57.055416 21481 leader_election.cc:302] T b9d45300961942b281d8403e89e8a4f8 P 4dc6fe52db3c4e3983ca92bca03b4bbe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1004ea961b344f06b6a5033e2c069359, 4dc6fe52db3c4e3983ca92bca03b4bbe; no voters:
I20260929 18:37:57.055598 21771 raft_consensus.cc:2781] T b9d45300961942b281d8403e89e8a4f8 P 4dc6fe52db3c4e3983ca92bca03b4bbe [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:37:57.055801 21771 raft_consensus.cc:686] T b9d45300961942b281d8403e89e8a4f8 P 4dc6fe52db3c4e3983ca92bca03b4bbe [term 1 LEADER]: Becoming Leader. State: Replica: 4dc6fe52db3c4e3983ca92bca03b4bbe, State: Running, Role: LEADER
I20260929 18:37:57.056082 21771 consensus_queue.cc:237] T b9d45300961942b281d8403e89e8a4f8 P 4dc6fe52db3c4e3983ca92bca03b4bbe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1004ea961b344f06b6a5033e2c069359" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 45113 } } peers { permanent_uuid: "4dc6fe52db3c4e3983ca92bca03b4bbe" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 37145 } } peers { permanent_uuid: "9741dfa235754052b0b0c2acc05483e6" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 44055 } }
I20260929 18:37:57.057346 21809 tablet_replica.cc:439] T b9d45300961942b281d8403e89e8a4f8 P 4dc6fe52db3c4e3983ca92bca03b4bbe: TxnStatusTablet state changed. Reason: New leader 4dc6fe52db3c4e3983ca92bca03b4bbe. Latest consensus state: current_term: 1 leader_uuid: "4dc6fe52db3c4e3983ca92bca03b4bbe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1004ea961b344f06b6a5033e2c069359" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 45113 } } peers { permanent_uuid: "4dc6fe52db3c4e3983ca92bca03b4bbe" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 37145 } } peers { permanent_uuid: "9741dfa235754052b0b0c2acc05483e6" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 44055 } } }
I20260929 18:37:57.057476 21809 tablet_replica.cc:442] T b9d45300961942b281d8403e89e8a4f8 P 4dc6fe52db3c4e3983ca92bca03b4bbe: This TxnStatusTablet replica's current role is: LEADER
I20260929 18:37:57.057723 21811 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 18:37:57.057874 21811 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 18:37:57.058521 21251 catalog_manager.cc:5495] T b9d45300961942b281d8403e89e8a4f8 P 4dc6fe52db3c4e3983ca92bca03b4bbe reported cstate change: term changed from 0 to 1, leader changed from <none> to 4dc6fe52db3c4e3983ca92bca03b4bbe (127.13.100.2). New cstate: current_term: 1 leader_uuid: "4dc6fe52db3c4e3983ca92bca03b4bbe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1004ea961b344f06b6a5033e2c069359" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 45113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4dc6fe52db3c4e3983ca92bca03b4bbe" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 37145 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9741dfa235754052b0b0c2acc05483e6" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 44055 } health_report { overall_health: UNKNOWN } } }
I20260929 18:37:57.071703 21802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:57.068048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.2:37145 (local address 127.0.0.1:50090)
0929 18:37:57.068214 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:57.068230 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:57.068380 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:57.068695 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:57.068698 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:57.068706 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 18:37:57.068811 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:57.068816 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:57.069543 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:57.069547 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:57.070219 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:57.070228 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:57.070666 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:57.070669 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:57.070757 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:57.070796 (+ 39us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 18:37:57.071422 (+ 626us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 18:37:57.071437 (+ 15us) client_negotiation.cc:769] Sending connection context
0929 18:37:57.071514 (+ 77us) client_negotiation.cc:240] Negotiation successful
0929 18:37:57.071593 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260929 18:37:57.071697 21759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:57.068133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50090 (local address 127.13.100.2:37145)
0929 18:37:57.068239 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:57.068246 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:57.068312 (+ 66us) server_negotiation.cc:407] Connection header received
0929 18:37:57.068512 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:57.068523 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:57.068573 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:57.068669 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:37:57.068915 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:57.069427 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:57.070334 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:57.070555 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:57.070669 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:57.070949 (+ 280us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:37:57.071327 (+ 378us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:37:57.071444 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:57.071514 (+ 70us) server_negotiation.cc:299] Negotiation successful
0929 18:37:57.071564 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 18:37:57.074654 21409 raft_consensus.cc:1261] T b9d45300961942b281d8403e89e8a4f8 P 1004ea961b344f06b6a5033e2c069359 [term 1 FOLLOWER]: Refusing update from remote peer 4dc6fe52db3c4e3983ca92bca03b4bbe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:37:57.075155 21689 raft_consensus.cc:1261] T b9d45300961942b281d8403e89e8a4f8 P 9741dfa235754052b0b0c2acc05483e6 [term 1 FOLLOWER]: Refusing update from remote peer 4dc6fe52db3c4e3983ca92bca03b4bbe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:37:57.075402 21771 consensus_queue.cc:1035] T b9d45300961942b281d8403e89e8a4f8 P 4dc6fe52db3c4e3983ca92bca03b4bbe [LEADER]: Connected to new peer: Peer: permanent_uuid: "1004ea961b344f06b6a5033e2c069359" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 45113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:37:57.075589 21809 consensus_queue.cc:1035] T b9d45300961942b281d8403e89e8a4f8 P 4dc6fe52db3c4e3983ca92bca03b4bbe [LEADER]: Connected to new peer: Peer: permanent_uuid: "9741dfa235754052b0b0c2acc05483e6" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 44055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:37:57.077006 21769 tablet_replica.cc:439] T b9d45300961942b281d8403e89e8a4f8 P 9741dfa235754052b0b0c2acc05483e6: TxnStatusTablet state changed. Reason: New leader 4dc6fe52db3c4e3983ca92bca03b4bbe. Latest consensus state: current_term: 1 leader_uuid: "4dc6fe52db3c4e3983ca92bca03b4bbe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1004ea961b344f06b6a5033e2c069359" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 45113 } } peers { permanent_uuid: "4dc6fe52db3c4e3983ca92bca03b4bbe" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 37145 } } peers { permanent_uuid: "9741dfa235754052b0b0c2acc05483e6" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 44055 } } }
I20260929 18:37:57.077121 21769 tablet_replica.cc:442] T b9d45300961942b281d8403e89e8a4f8 P 9741dfa235754052b0b0c2acc05483e6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 18:37:57.077078 21774 tablet_replica.cc:439] T b9d45300961942b281d8403e89e8a4f8 P 1004ea961b344f06b6a5033e2c069359: TxnStatusTablet state changed. Reason: New leader 4dc6fe52db3c4e3983ca92bca03b4bbe. Latest consensus state: current_term: 1 leader_uuid: "4dc6fe52db3c4e3983ca92bca03b4bbe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1004ea961b344f06b6a5033e2c069359" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 45113 } } peers { permanent_uuid: "4dc6fe52db3c4e3983ca92bca03b4bbe" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 37145 } } peers { permanent_uuid: "9741dfa235754052b0b0c2acc05483e6" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 44055 } } }
I20260929 18:37:57.077189 21774 tablet_replica.cc:442] T b9d45300961942b281d8403e89e8a4f8 P 1004ea961b344f06b6a5033e2c069359: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 18:37:57.078720 21771 tablet_replica.cc:439] T b9d45300961942b281d8403e89e8a4f8 P 4dc6fe52db3c4e3983ca92bca03b4bbe: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4dc6fe52db3c4e3983ca92bca03b4bbe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1004ea961b344f06b6a5033e2c069359" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 45113 } } peers { permanent_uuid: "4dc6fe52db3c4e3983ca92bca03b4bbe" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 37145 } } peers { permanent_uuid: "9741dfa235754052b0b0c2acc05483e6" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 44055 } } }
I20260929 18:37:57.078819 21771 tablet_replica.cc:442] T b9d45300961942b281d8403e89e8a4f8 P 4dc6fe52db3c4e3983ca92bca03b4bbe: This TxnStatusTablet replica's current role is: LEADER
I20260929 18:37:57.080470 21779 mvcc.cc:205] Tried to move back new op lower bound from 7334736187691687936 to 7334736187624427520. Current Snapshot: MvccSnapshot[applied={T|T < 7334736187691687936}]
I20260929 18:37:57.080647 21769 tablet_replica.cc:439] T b9d45300961942b281d8403e89e8a4f8 P 9741dfa235754052b0b0c2acc05483e6: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4dc6fe52db3c4e3983ca92bca03b4bbe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1004ea961b344f06b6a5033e2c069359" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 45113 } } peers { permanent_uuid: "4dc6fe52db3c4e3983ca92bca03b4bbe" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 37145 } } peers { permanent_uuid: "9741dfa235754052b0b0c2acc05483e6" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 44055 } } }
I20260929 18:37:57.080744 21769 tablet_replica.cc:442] T b9d45300961942b281d8403e89e8a4f8 P 9741dfa235754052b0b0c2acc05483e6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 18:37:57.081332 21809 tablet_replica.cc:439] T b9d45300961942b281d8403e89e8a4f8 P 4dc6fe52db3c4e3983ca92bca03b4bbe: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4dc6fe52db3c4e3983ca92bca03b4bbe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1004ea961b344f06b6a5033e2c069359" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 45113 } } peers { permanent_uuid: "4dc6fe52db3c4e3983ca92bca03b4bbe" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 37145 } } peers { permanent_uuid: "9741dfa235754052b0b0c2acc05483e6" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 44055 } } }
I20260929 18:37:57.081441 21809 tablet_replica.cc:442] T b9d45300961942b281d8403e89e8a4f8 P 4dc6fe52db3c4e3983ca92bca03b4bbe: This TxnStatusTablet replica's current role is: LEADER
I20260929 18:37:57.082891 21774 tablet_replica.cc:439] T b9d45300961942b281d8403e89e8a4f8 P 1004ea961b344f06b6a5033e2c069359: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4dc6fe52db3c4e3983ca92bca03b4bbe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1004ea961b344f06b6a5033e2c069359" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 45113 } } peers { permanent_uuid: "4dc6fe52db3c4e3983ca92bca03b4bbe" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 37145 } } peers { permanent_uuid: "9741dfa235754052b0b0c2acc05483e6" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 44055 } } }
I20260929 18:37:57.083001 21774 tablet_replica.cc:442] T b9d45300961942b281d8403e89e8a4f8 P 1004ea961b344f06b6a5033e2c069359: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 18:37:57.087234 21762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:57.083384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48710 (local address 127.13.100.3:44055)
0929 18:37:57.083510 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:57.083515 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:57.083530 (+ 15us) server_negotiation.cc:407] Connection header received
0929 18:37:57.083600 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:57.083603 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:57.083668 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:57.083767 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:37:57.084056 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:57.084764 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:57.085673 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:57.085960 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:57.086070 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:57.086323 (+ 253us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:37:57.086812 (+ 489us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:37:57.086929 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:57.087021 (+ 92us) server_negotiation.cc:299] Negotiation successful
0929 18:37:57.087078 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 18:37:57.097703 21818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:57.093974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.2:37145 (local address 127.13.100.3:45179)
0929 18:37:57.094262 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:57.094273 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:57.094410 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:57.094664 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:57.094667 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:57.094674 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 18:37:57.095119 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:57.095125 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:57.095847 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:57.095852 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:57.096453 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:57.096460 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:57.096952 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:57.096956 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:57.097045 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:57.097084 (+ 39us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 18:37:57.097381 (+ 297us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 18:37:57.097388 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 18:37:57.097467 (+ 79us) client_negotiation.cc:240] Negotiation successful
0929 18:37:57.097542 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":93,"threads_started":1}
I20260929 18:37:57.097690 21759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:57.094068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.3:45179 (local address 127.13.100.2:37145)
0929 18:37:57.094202 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:57.094206 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:57.094340 (+ 134us) server_negotiation.cc:407] Connection header received
0929 18:37:57.094517 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:57.094520 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:57.094570 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:57.094646 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:37:57.095229 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:57.095738 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:57.096593 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:57.096828 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:57.096944 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:57.097219 (+ 275us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:37:57.097290 (+ 71us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:37:57.097367 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:57.097488 (+ 121us) server_negotiation.cc:299] Negotiation successful
0929 18:37:57.097543 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 18:37:57.103982 21549 raft_consensus.cc:1261] T f21fff2ca04148d19f65024180143c61 P 4dc6fe52db3c4e3983ca92bca03b4bbe [term 1 FOLLOWER]: Refusing update from remote peer 9741dfa235754052b0b0c2acc05483e6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:37:57.104295 21409 raft_consensus.cc:1261] T f21fff2ca04148d19f65024180143c61 P 1004ea961b344f06b6a5033e2c069359 [term 1 FOLLOWER]: Refusing update from remote peer 9741dfa235754052b0b0c2acc05483e6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:37:57.104415 21795 consensus_queue.cc:1035] T f21fff2ca04148d19f65024180143c61 P 9741dfa235754052b0b0c2acc05483e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4dc6fe52db3c4e3983ca92bca03b4bbe" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 37145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:37:57.104663 21769 consensus_queue.cc:1035] T f21fff2ca04148d19f65024180143c61 P 9741dfa235754052b0b0c2acc05483e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1004ea961b344f06b6a5033e2c069359" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 45113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:37:57.125639 21762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:57.122050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.2:40655 (local address 127.13.100.3:44055)
0929 18:37:57.122166 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:57.122171 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:57.122389 (+ 218us) server_negotiation.cc:407] Connection header received
0929 18:37:57.122560 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:57.122564 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:57.122614 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:57.122698 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:37:57.122956 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:57.123475 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:57.124674 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:57.124882 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:57.124954 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:57.125262 (+ 308us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:37:57.125323 (+ 61us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:37:57.125376 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:57.125491 (+ 115us) server_negotiation.cc:299] Negotiation successful
0929 18:37:57.125538 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260929 18:37:57.125723 21822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:57.122017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.3:44055 (local address 127.13.100.2:40655)
0929 18:37:57.122305 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:57.122319 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:57.122459 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:57.122716 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:57.122719 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:57.122726 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 18:37:57.122852 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:57.122858 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:57.123581 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:57.123584 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:57.124552 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:57.124562 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:57.124980 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:57.124983 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:57.125073 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:57.125122 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 18:37:57.125403 (+ 281us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 18:37:57.125409 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 18:37:57.125476 (+ 67us) client_negotiation.cc:240] Negotiation successful
0929 18:37:57.125551 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":97,"threads_started":1}
I20260929 18:37:57.151894 21251 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59372:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:37:57.152060 21251 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59372:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:37:57.154078 21251 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 73570d97b6164fdb8636f3f3c01b24e4: Sending DeleteTablet for 3 replicas of tablet f21fff2ca04148d19f65024180143c61
I20260929 18:37:57.154681 21389 tablet_service.cc:1514] Processing DeleteTablet for tablet f21fff2ca04148d19f65024180143c61 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:37:57 UTC) from {username='oryx', principal='oryx/127.13.100.62@KRBTEST.COM'} at 127.0.0.1:39396
I20260929 18:37:57.154747 21669 tablet_service.cc:1514] Processing DeleteTablet for tablet f21fff2ca04148d19f65024180143c61 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:37:57 UTC) from {username='oryx', principal='oryx/127.13.100.62@KRBTEST.COM'} at 127.0.0.1:48708
I20260929 18:37:57.154860 21788 tablet_replica.cc:330] T f21fff2ca04148d19f65024180143c61 P 1004ea961b344f06b6a5033e2c069359: stopping tablet replica
I20260929 18:37:57.154913 21789 tablet_replica.cc:330] T f21fff2ca04148d19f65024180143c61 P 9741dfa235754052b0b0c2acc05483e6: stopping tablet replica
I20260929 18:37:57.154992 21788 raft_consensus.cc:2229] T f21fff2ca04148d19f65024180143c61 P 1004ea961b344f06b6a5033e2c069359 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:37:57.155036 21789 raft_consensus.cc:2229] T f21fff2ca04148d19f65024180143c61 P 9741dfa235754052b0b0c2acc05483e6 [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:37:57.155112 21788 raft_consensus.cc:2258] T f21fff2ca04148d19f65024180143c61 P 1004ea961b344f06b6a5033e2c069359 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:37:57.155329 21789 raft_consensus.cc:2258] T f21fff2ca04148d19f65024180143c61 P 9741dfa235754052b0b0c2acc05483e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:37:57.155864 21529 tablet_service.cc:1514] Processing DeleteTablet for tablet f21fff2ca04148d19f65024180143c61 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:37:57 UTC) from {username='oryx', principal='oryx/127.13.100.62@KRBTEST.COM'} at 127.0.0.1:50074
I20260929 18:37:57.155946 21789 ts_tablet_manager.cc:1905] T f21fff2ca04148d19f65024180143c61 P 9741dfa235754052b0b0c2acc05483e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:37:57.155946 21788 ts_tablet_manager.cc:1905] T f21fff2ca04148d19f65024180143c61 P 1004ea961b344f06b6a5033e2c069359: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:37:57.156045 21787 tablet_replica.cc:330] T f21fff2ca04148d19f65024180143c61 P 4dc6fe52db3c4e3983ca92bca03b4bbe: stopping tablet replica
I20260929 18:37:57.156138 21787 raft_consensus.cc:2229] T f21fff2ca04148d19f65024180143c61 P 4dc6fe52db3c4e3983ca92bca03b4bbe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:37:57.156234 21787 raft_consensus.cc:2258] T f21fff2ca04148d19f65024180143c61 P 4dc6fe52db3c4e3983ca92bca03b4bbe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:37:57.158679 21789 ts_tablet_manager.cc:1918] T f21fff2ca04148d19f65024180143c61 P 9741dfa235754052b0b0c2acc05483e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 18:37:57.158679 21788 ts_tablet_manager.cc:1918] T f21fff2ca04148d19f65024180143c61 P 1004ea961b344f06b6a5033e2c069359: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 18:37:57.158746 21789 log.cc:1196] T f21fff2ca04148d19f65024180143c61 P 9741dfa235754052b0b0c2acc05483e6: Deleting WAL directory at /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707036041078-13712-0/minicluster-data/ts-2/wal/wals/f21fff2ca04148d19f65024180143c61
I20260929 18:37:57.158746 21788 log.cc:1196] T f21fff2ca04148d19f65024180143c61 P 1004ea961b344f06b6a5033e2c069359: Deleting WAL directory at /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707036041078-13712-0/minicluster-data/ts-0/wal/wals/f21fff2ca04148d19f65024180143c61
I20260929 18:37:57.159005 21788 ts_tablet_manager.cc:1939] T f21fff2ca04148d19f65024180143c61 P 1004ea961b344f06b6a5033e2c069359: Deleting consensus metadata
I20260929 18:37:57.159068 21789 ts_tablet_manager.cc:1939] T f21fff2ca04148d19f65024180143c61 P 9741dfa235754052b0b0c2acc05483e6: Deleting consensus metadata
I20260929 18:37:57.159782 21238 catalog_manager.cc:4841] TS 9741dfa235754052b0b0c2acc05483e6 (127.13.100.3:44055): tablet f21fff2ca04148d19f65024180143c61 (table test-table [id=90cdb37bd9fa4ef1923c6774af0498cf]) successfully deleted
I20260929 18:37:57.159842 21237 catalog_manager.cc:4841] TS 1004ea961b344f06b6a5033e2c069359 (127.13.100.1:45113): tablet f21fff2ca04148d19f65024180143c61 (table test-table [id=90cdb37bd9fa4ef1923c6774af0498cf]) successfully deleted
I20260929 18:37:57.160492 21787 ts_tablet_manager.cc:1905] T f21fff2ca04148d19f65024180143c61 P 4dc6fe52db3c4e3983ca92bca03b4bbe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:37:57.162715 21787 ts_tablet_manager.cc:1918] T f21fff2ca04148d19f65024180143c61 P 4dc6fe52db3c4e3983ca92bca03b4bbe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 18:37:57.162810 21787 log.cc:1196] T f21fff2ca04148d19f65024180143c61 P 4dc6fe52db3c4e3983ca92bca03b4bbe: Deleting WAL directory at /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707036041078-13712-0/minicluster-data/ts-1/wal/wals/f21fff2ca04148d19f65024180143c61
I20260929 18:37:57.163045 21787 ts_tablet_manager.cc:1939] T f21fff2ca04148d19f65024180143c61 P 4dc6fe52db3c4e3983ca92bca03b4bbe: Deleting consensus metadata
I20260929 18:37:57.163717 21238 catalog_manager.cc:4841] TS 4dc6fe52db3c4e3983ca92bca03b4bbe (127.13.100.2:37145): tablet f21fff2ca04148d19f65024180143c61 (table test-table [id=90cdb37bd9fa4ef1923c6774af0498cf]) successfully deleted
Sep 29 18:37:57 dist-test-slave-j1xs krb5kdc[21207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707077, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 18:37:57 dist-test-slave-j1xs krb5kdc[21207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.13.100.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:37:57 dist-test-slave-j1xs krb5kdc[21207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.13.100.62@KRBTEST.COM, Server not found in Kerberos database
W20260929 18:37:57.186084 21803 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:37:57.179416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59382 (local address 127.13.100.62:39731)
0929 18:37:57.179562 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:57.179569 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:57.179833 (+ 264us) server_negotiation.cc:407] Connection header received
0929 18:37:57.180005 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:57.180007 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:57.180054 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:57.180140 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:57.180827 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:57.181356 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:57.182500 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:57.182739 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:57.182857 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:57.185985 (+ 3128us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59382: BlockingRecv error: recv got EOF from 127.0.0.1:59382 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 29 18:37:57 dist-test-slave-j1xs krb5kdc[21207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707077, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.13.100.62@KRBTEST.COM
I20260929 18:37:57.196662 21803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:57.188087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59394 (local address 127.13.100.62:39731)
0929 18:37:57.188198 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:57.188202 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:57.188567 (+ 365us) server_negotiation.cc:407] Connection header received
0929 18:37:57.188686 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:57.188689 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:57.188738 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:57.188821 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:57.189418 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:57.190087 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:57.190877 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:57.191101 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:57.191205 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:57.193529 (+ 2324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:57.193533 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:57.193536 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:37:57.193567 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:57.195028 (+ 1461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:57.195446 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:57.195451 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:57.195453 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:57.195531 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:57.195784 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:57.195788 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:57.195790 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:57.195964 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:57.196100 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:57.196360 (+ 260us) server_negotiation.cc:299] Negotiation successful
0929 18:37:57.196483 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 18:37:57.199272 21251 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59394:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:37:57.199719 21251 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:37:57.204499 21389 tablet_service.cc:1467] Processing CreateTablet for tablet 136a10fa159f42bda9a7d8eb19b811c4 (DEFAULT_TABLE table=test-table [id=03181ac04c81459f937e0a2cadba8b44]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:37:57.204799 21389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 136a10fa159f42bda9a7d8eb19b811c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:37:57.206802 21765 tablet_bootstrap.cc:492] T 136a10fa159f42bda9a7d8eb19b811c4 P 1004ea961b344f06b6a5033e2c069359: Bootstrap starting.
I20260929 18:37:57.207376 21669 tablet_service.cc:1467] Processing CreateTablet for tablet 136a10fa159f42bda9a7d8eb19b811c4 (DEFAULT_TABLE table=test-table [id=03181ac04c81459f937e0a2cadba8b44]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:37:57.207644 21669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 136a10fa159f42bda9a7d8eb19b811c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:37:57.207690 21765 tablet_bootstrap.cc:654] T 136a10fa159f42bda9a7d8eb19b811c4 P 1004ea961b344f06b6a5033e2c069359: Neither blocks nor log segments found. Creating new log.
I20260929 18:37:57.209554 21763 tablet_bootstrap.cc:492] T 136a10fa159f42bda9a7d8eb19b811c4 P 9741dfa235754052b0b0c2acc05483e6: Bootstrap starting.
I20260929 18:37:57.209990 21529 tablet_service.cc:1467] Processing CreateTablet for tablet 136a10fa159f42bda9a7d8eb19b811c4 (DEFAULT_TABLE table=test-table [id=03181ac04c81459f937e0a2cadba8b44]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:37:57.210276 21529 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 136a10fa159f42bda9a7d8eb19b811c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:37:57.211591 21763 tablet_bootstrap.cc:654] T 136a10fa159f42bda9a7d8eb19b811c4 P 9741dfa235754052b0b0c2acc05483e6: Neither blocks nor log segments found. Creating new log.
I20260929 18:37:57.212798 21763 tablet_bootstrap.cc:492] T 136a10fa159f42bda9a7d8eb19b811c4 P 9741dfa235754052b0b0c2acc05483e6: No bootstrap required, opened a new log
I20260929 18:37:57.212821 21764 tablet_bootstrap.cc:492] T 136a10fa159f42bda9a7d8eb19b811c4 P 4dc6fe52db3c4e3983ca92bca03b4bbe: Bootstrap starting.
I20260929 18:37:57.212895 21763 ts_tablet_manager.cc:1397] T 136a10fa159f42bda9a7d8eb19b811c4 P 9741dfa235754052b0b0c2acc05483e6: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 18:37:57.213325 21763 raft_consensus.cc:348] T 136a10fa159f42bda9a7d8eb19b811c4 P 9741dfa235754052b0b0c2acc05483e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1004ea961b344f06b6a5033e2c069359" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 45113 } } peers { permanent_uuid: "4dc6fe52db3c4e3983ca92bca03b4bbe" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 37145 } } peers { permanent_uuid: "9741dfa235754052b0b0c2acc05483e6" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 44055 } }
I20260929 18:37:57.213430 21763 raft_consensus.cc:374] T 136a10fa159f42bda9a7d8eb19b811c4 P 9741dfa235754052b0b0c2acc05483e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:37:57.213462 21763 raft_consensus.cc:729] T 136a10fa159f42bda9a7d8eb19b811c4 P 9741dfa235754052b0b0c2acc05483e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9741dfa235754052b0b0c2acc05483e6, State: Initialized, Role: FOLLOWER
I20260929 18:37:57.213570 21763 consensus_queue.cc:260] T 136a10fa159f42bda9a7d8eb19b811c4 P 9741dfa235754052b0b0c2acc05483e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1004ea961b344f06b6a5033e2c069359" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 45113 } } peers { permanent_uuid: "4dc6fe52db3c4e3983ca92bca03b4bbe" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 37145 } } peers { permanent_uuid: "9741dfa235754052b0b0c2acc05483e6" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 44055 } }
I20260929 18:37:57.213675 21764 tablet_bootstrap.cc:654] T 136a10fa159f42bda9a7d8eb19b811c4 P 4dc6fe52db3c4e3983ca92bca03b4bbe: Neither blocks nor log segments found. Creating new log.
I20260929 18:37:57.213929 21763 ts_tablet_manager.cc:1428] T 136a10fa159f42bda9a7d8eb19b811c4 P 9741dfa235754052b0b0c2acc05483e6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 18:37:57.215204 21764 tablet_bootstrap.cc:492] T 136a10fa159f42bda9a7d8eb19b811c4 P 4dc6fe52db3c4e3983ca92bca03b4bbe: No bootstrap required, opened a new log
I20260929 18:37:57.215296 21764 ts_tablet_manager.cc:1397] T 136a10fa159f42bda9a7d8eb19b811c4 P 4dc6fe52db3c4e3983ca92bca03b4bbe: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260929 18:37:57.215752 21764 raft_consensus.cc:348] T 136a10fa159f42bda9a7d8eb19b811c4 P 4dc6fe52db3c4e3983ca92bca03b4bbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1004ea961b344f06b6a5033e2c069359" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 45113 } } peers { permanent_uuid: "4dc6fe52db3c4e3983ca92bca03b4bbe" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 37145 } } peers { permanent_uuid: "9741dfa235754052b0b0c2acc05483e6" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 44055 } }
I20260929 18:37:57.215859 21764 raft_consensus.cc:374] T 136a10fa159f42bda9a7d8eb19b811c4 P 4dc6fe52db3c4e3983ca92bca03b4bbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:37:57.215893 21764 raft_consensus.cc:729] T 136a10fa159f42bda9a7d8eb19b811c4 P 4dc6fe52db3c4e3983ca92bca03b4bbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4dc6fe52db3c4e3983ca92bca03b4bbe, State: Initialized, Role: FOLLOWER
I20260929 18:37:57.216003 21764 consensus_queue.cc:260] T 136a10fa159f42bda9a7d8eb19b811c4 P 4dc6fe52db3c4e3983ca92bca03b4bbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1004ea961b344f06b6a5033e2c069359" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 45113 } } peers { permanent_uuid: "4dc6fe52db3c4e3983ca92bca03b4bbe" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 37145 } } peers { permanent_uuid: "9741dfa235754052b0b0c2acc05483e6" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 44055 } }
I20260929 18:37:57.216274 21764 ts_tablet_manager.cc:1428] T 136a10fa159f42bda9a7d8eb19b811c4 P 4dc6fe52db3c4e3983ca92bca03b4bbe: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 18:37:57.218500 21765 tablet_bootstrap.cc:492] T 136a10fa159f42bda9a7d8eb19b811c4 P 1004ea961b344f06b6a5033e2c069359: No bootstrap required, opened a new log
I20260929 18:37:57.218613 21765 ts_tablet_manager.cc:1397] T 136a10fa159f42bda9a7d8eb19b811c4 P 1004ea961b344f06b6a5033e2c069359: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.000s
I20260929 18:37:57.218976 21765 raft_consensus.cc:348] T 136a10fa159f42bda9a7d8eb19b811c4 P 1004ea961b344f06b6a5033e2c069359 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1004ea961b344f06b6a5033e2c069359" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 45113 } } peers { permanent_uuid: "4dc6fe52db3c4e3983ca92bca03b4bbe" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 37145 } } peers { permanent_uuid: "9741dfa235754052b0b0c2acc05483e6" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 44055 } }
I20260929 18:37:57.219123 21765 raft_consensus.cc:374] T 136a10fa159f42bda9a7d8eb19b811c4 P 1004ea961b344f06b6a5033e2c069359 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:37:57.219161 21765 raft_consensus.cc:729] T 136a10fa159f42bda9a7d8eb19b811c4 P 1004ea961b344f06b6a5033e2c069359 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1004ea961b344f06b6a5033e2c069359, State: Initialized, Role: FOLLOWER
I20260929 18:37:57.219291 21765 consensus_queue.cc:260] T 136a10fa159f42bda9a7d8eb19b811c4 P 1004ea961b344f06b6a5033e2c069359 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1004ea961b344f06b6a5033e2c069359" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 45113 } } peers { permanent_uuid: "4dc6fe52db3c4e3983ca92bca03b4bbe" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 37145 } } peers { permanent_uuid: "9741dfa235754052b0b0c2acc05483e6" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 44055 } }
I20260929 18:37:57.219538 21765 ts_tablet_manager.cc:1428] T 136a10fa159f42bda9a7d8eb19b811c4 P 1004ea961b344f06b6a5033e2c069359: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 18:37:57.273001 21809 raft_consensus.cc:482] T 136a10fa159f42bda9a7d8eb19b811c4 P 4dc6fe52db3c4e3983ca92bca03b4bbe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:37:57.273159 21809 raft_consensus.cc:504] T 136a10fa159f42bda9a7d8eb19b811c4 P 4dc6fe52db3c4e3983ca92bca03b4bbe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1004ea961b344f06b6a5033e2c069359" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 45113 } } peers { permanent_uuid: "4dc6fe52db3c4e3983ca92bca03b4bbe" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 37145 } } peers { permanent_uuid: "9741dfa235754052b0b0c2acc05483e6" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 44055 } }
I20260929 18:37:57.273609 21809 leader_election.cc:288] T 136a10fa159f42bda9a7d8eb19b811c4 P 4dc6fe52db3c4e3983ca92bca03b4bbe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1004ea961b344f06b6a5033e2c069359 (127.13.100.1:45113), 9741dfa235754052b0b0c2acc05483e6 (127.13.100.3:44055)
I20260929 18:37:57.273995 21409 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "136a10fa159f42bda9a7d8eb19b811c4" candidate_uuid: "4dc6fe52db3c4e3983ca92bca03b4bbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1004ea961b344f06b6a5033e2c069359" is_pre_election: true
I20260929 18:37:57.274056 21689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "136a10fa159f42bda9a7d8eb19b811c4" candidate_uuid: "4dc6fe52db3c4e3983ca92bca03b4bbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9741dfa235754052b0b0c2acc05483e6" is_pre_election: true
I20260929 18:37:57.274134 21409 raft_consensus.cc:2454] T 136a10fa159f42bda9a7d8eb19b811c4 P 1004ea961b344f06b6a5033e2c069359 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4dc6fe52db3c4e3983ca92bca03b4bbe in term 0.
I20260929 18:37:57.274206 21689 raft_consensus.cc:2454] T 136a10fa159f42bda9a7d8eb19b811c4 P 9741dfa235754052b0b0c2acc05483e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4dc6fe52db3c4e3983ca92bca03b4bbe in term 0.
I20260929 18:37:57.274474 21481 leader_election.cc:302] T 136a10fa159f42bda9a7d8eb19b811c4 P 4dc6fe52db3c4e3983ca92bca03b4bbe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1004ea961b344f06b6a5033e2c069359, 4dc6fe52db3c4e3983ca92bca03b4bbe; no voters:
I20260929 18:37:57.274674 21809 raft_consensus.cc:2781] T 136a10fa159f42bda9a7d8eb19b811c4 P 4dc6fe52db3c4e3983ca92bca03b4bbe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:37:57.274735 21809 raft_consensus.cc:482] T 136a10fa159f42bda9a7d8eb19b811c4 P 4dc6fe52db3c4e3983ca92bca03b4bbe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:37:57.274777 21809 raft_consensus.cc:3037] T 136a10fa159f42bda9a7d8eb19b811c4 P 4dc6fe52db3c4e3983ca92bca03b4bbe [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:37:57.275760 21809 raft_consensus.cc:504] T 136a10fa159f42bda9a7d8eb19b811c4 P 4dc6fe52db3c4e3983ca92bca03b4bbe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1004ea961b344f06b6a5033e2c069359" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 45113 } } peers { permanent_uuid: "4dc6fe52db3c4e3983ca92bca03b4bbe" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 37145 } } peers { permanent_uuid: "9741dfa235754052b0b0c2acc05483e6" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 44055 } }
I20260929 18:37:57.276167 21809 leader_election.cc:288] T 136a10fa159f42bda9a7d8eb19b811c4 P 4dc6fe52db3c4e3983ca92bca03b4bbe [CANDIDATE]: Term 1 election: Requested vote from peers 1004ea961b344f06b6a5033e2c069359 (127.13.100.1:45113), 9741dfa235754052b0b0c2acc05483e6 (127.13.100.3:44055)
I20260929 18:37:57.276460 21409 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "136a10fa159f42bda9a7d8eb19b811c4" candidate_uuid: "4dc6fe52db3c4e3983ca92bca03b4bbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1004ea961b344f06b6a5033e2c069359"
I20260929 18:37:57.276518 21689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "136a10fa159f42bda9a7d8eb19b811c4" candidate_uuid: "4dc6fe52db3c4e3983ca92bca03b4bbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9741dfa235754052b0b0c2acc05483e6"
I20260929 18:37:57.276612 21689 raft_consensus.cc:3037] T 136a10fa159f42bda9a7d8eb19b811c4 P 9741dfa235754052b0b0c2acc05483e6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:37:57.276607 21409 raft_consensus.cc:3037] T 136a10fa159f42bda9a7d8eb19b811c4 P 1004ea961b344f06b6a5033e2c069359 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:37:57.277673 21689 raft_consensus.cc:2454] T 136a10fa159f42bda9a7d8eb19b811c4 P 9741dfa235754052b0b0c2acc05483e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4dc6fe52db3c4e3983ca92bca03b4bbe in term 1.
I20260929 18:37:57.277673 21409 raft_consensus.cc:2454] T 136a10fa159f42bda9a7d8eb19b811c4 P 1004ea961b344f06b6a5033e2c069359 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4dc6fe52db3c4e3983ca92bca03b4bbe in term 1.
I20260929 18:37:57.277997 21482 leader_election.cc:302] T 136a10fa159f42bda9a7d8eb19b811c4 P 4dc6fe52db3c4e3983ca92bca03b4bbe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4dc6fe52db3c4e3983ca92bca03b4bbe, 9741dfa235754052b0b0c2acc05483e6; no voters:
I20260929 18:37:57.278162 21809 raft_consensus.cc:2781] T 136a10fa159f42bda9a7d8eb19b811c4 P 4dc6fe52db3c4e3983ca92bca03b4bbe [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:37:57.278306 21809 raft_consensus.cc:686] T 136a10fa159f42bda9a7d8eb19b811c4 P 4dc6fe52db3c4e3983ca92bca03b4bbe [term 1 LEADER]: Becoming Leader. State: Replica: 4dc6fe52db3c4e3983ca92bca03b4bbe, State: Running, Role: LEADER
I20260929 18:37:57.278448 21809 consensus_queue.cc:237] T 136a10fa159f42bda9a7d8eb19b811c4 P 4dc6fe52db3c4e3983ca92bca03b4bbe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1004ea961b344f06b6a5033e2c069359" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 45113 } } peers { permanent_uuid: "4dc6fe52db3c4e3983ca92bca03b4bbe" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 37145 } } peers { permanent_uuid: "9741dfa235754052b0b0c2acc05483e6" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 44055 } }
I20260929 18:37:57.280256 21251 catalog_manager.cc:5495] T 136a10fa159f42bda9a7d8eb19b811c4 P 4dc6fe52db3c4e3983ca92bca03b4bbe reported cstate change: term changed from 0 to 1, leader changed from <none> to 4dc6fe52db3c4e3983ca92bca03b4bbe (127.13.100.2). New cstate: current_term: 1 leader_uuid: "4dc6fe52db3c4e3983ca92bca03b4bbe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1004ea961b344f06b6a5033e2c069359" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 45113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4dc6fe52db3c4e3983ca92bca03b4bbe" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 37145 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9741dfa235754052b0b0c2acc05483e6" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 44055 } health_report { overall_health: UNKNOWN } } }
I20260929 18:37:57.304893 21759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:57.301004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50100 (local address 127.13.100.2:37145)
0929 18:37:57.301129 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:57.301132 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:57.301197 (+ 65us) server_negotiation.cc:407] Connection header received
0929 18:37:57.301339 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:57.301341 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:57.301389 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:57.301465 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:37:57.301737 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:57.302459 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:57.303660 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:57.303909 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:57.303991 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:57.304357 (+ 366us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:37:57.304435 (+ 78us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:37:57.304511 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:57.304649 (+ 138us) server_negotiation.cc:299] Negotiation successful
0929 18:37:57.304700 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 18:37:57.307271 21689 raft_consensus.cc:1261] T 136a10fa159f42bda9a7d8eb19b811c4 P 9741dfa235754052b0b0c2acc05483e6 [term 1 FOLLOWER]: Refusing update from remote peer 4dc6fe52db3c4e3983ca92bca03b4bbe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:37:57.307377 21409 raft_consensus.cc:1261] T 136a10fa159f42bda9a7d8eb19b811c4 P 1004ea961b344f06b6a5033e2c069359 [term 1 FOLLOWER]: Refusing update from remote peer 4dc6fe52db3c4e3983ca92bca03b4bbe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:37:57.307763 21809 consensus_queue.cc:1035] T 136a10fa159f42bda9a7d8eb19b811c4 P 4dc6fe52db3c4e3983ca92bca03b4bbe [LEADER]: Connected to new peer: Peer: permanent_uuid: "9741dfa235754052b0b0c2acc05483e6" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 44055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:37:57.307937 21771 consensus_queue.cc:1035] T 136a10fa159f42bda9a7d8eb19b811c4 P 4dc6fe52db3c4e3983ca92bca03b4bbe [LEADER]: Connected to new peer: Peer: permanent_uuid: "1004ea961b344f06b6a5033e2c069359" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 45113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:37:57.318982 21250 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59394:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:37:57.319145 21250 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59394:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:37:57.321354 21250 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 73570d97b6164fdb8636f3f3c01b24e4: Sending DeleteTablet for 3 replicas of tablet 136a10fa159f42bda9a7d8eb19b811c4
I20260929 18:37:57.321923 21389 tablet_service.cc:1514] Processing DeleteTablet for tablet 136a10fa159f42bda9a7d8eb19b811c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:37:57 UTC) from {username='oryx', principal='oryx/127.13.100.62@KRBTEST.COM'} at 127.0.0.1:39396
I20260929 18:37:57.322113 21788 tablet_replica.cc:330] T 136a10fa159f42bda9a7d8eb19b811c4 P 1004ea961b344f06b6a5033e2c069359: stopping tablet replica
I20260929 18:37:57.322201 21529 tablet_service.cc:1514] Processing DeleteTablet for tablet 136a10fa159f42bda9a7d8eb19b811c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:37:57 UTC) from {username='oryx', principal='oryx/127.13.100.62@KRBTEST.COM'} at 127.0.0.1:50074
I20260929 18:37:57.322232 21788 raft_consensus.cc:2229] T 136a10fa159f42bda9a7d8eb19b811c4 P 1004ea961b344f06b6a5033e2c069359 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:37:57.322348 21788 raft_consensus.cc:2258] T 136a10fa159f42bda9a7d8eb19b811c4 P 1004ea961b344f06b6a5033e2c069359 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:37:57.322361 21787 tablet_replica.cc:330] T 136a10fa159f42bda9a7d8eb19b811c4 P 4dc6fe52db3c4e3983ca92bca03b4bbe: stopping tablet replica
I20260929 18:37:57.322467 21787 raft_consensus.cc:2229] T 136a10fa159f42bda9a7d8eb19b811c4 P 4dc6fe52db3c4e3983ca92bca03b4bbe [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:37:57.322741 21787 raft_consensus.cc:2258] T 136a10fa159f42bda9a7d8eb19b811c4 P 4dc6fe52db3c4e3983ca92bca03b4bbe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:37:57.323145 21669 tablet_service.cc:1514] Processing DeleteTablet for tablet 136a10fa159f42bda9a7d8eb19b811c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:37:57 UTC) from {username='oryx', principal='oryx/127.13.100.62@KRBTEST.COM'} at 127.0.0.1:48708
I20260929 18:37:57.323323 21789 tablet_replica.cc:330] T 136a10fa159f42bda9a7d8eb19b811c4 P 9741dfa235754052b0b0c2acc05483e6: stopping tablet replica
I20260929 18:37:57.323449 21789 raft_consensus.cc:2229] T 136a10fa159f42bda9a7d8eb19b811c4 P 9741dfa235754052b0b0c2acc05483e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:37:57.323576 21789 raft_consensus.cc:2258] T 136a10fa159f42bda9a7d8eb19b811c4 P 9741dfa235754052b0b0c2acc05483e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:37:57.323925 21788 ts_tablet_manager.cc:1905] T 136a10fa159f42bda9a7d8eb19b811c4 P 1004ea961b344f06b6a5033e2c069359: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:37:57.324286 21787 ts_tablet_manager.cc:1905] T 136a10fa159f42bda9a7d8eb19b811c4 P 4dc6fe52db3c4e3983ca92bca03b4bbe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:37:57.326575 21788 ts_tablet_manager.cc:1918] T 136a10fa159f42bda9a7d8eb19b811c4 P 1004ea961b344f06b6a5033e2c069359: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:37:57.326702 21788 log.cc:1196] T 136a10fa159f42bda9a7d8eb19b811c4 P 1004ea961b344f06b6a5033e2c069359: Deleting WAL directory at /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707036041078-13712-0/minicluster-data/ts-0/wal/wals/136a10fa159f42bda9a7d8eb19b811c4
I20260929 18:37:57.326820 21787 ts_tablet_manager.cc:1918] T 136a10fa159f42bda9a7d8eb19b811c4 P 4dc6fe52db3c4e3983ca92bca03b4bbe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:37:57.326905 21787 log.cc:1196] T 136a10fa159f42bda9a7d8eb19b811c4 P 4dc6fe52db3c4e3983ca92bca03b4bbe: Deleting WAL directory at /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707036041078-13712-0/minicluster-data/ts-1/wal/wals/136a10fa159f42bda9a7d8eb19b811c4
I20260929 18:37:57.327183 21787 ts_tablet_manager.cc:1939] T 136a10fa159f42bda9a7d8eb19b811c4 P 4dc6fe52db3c4e3983ca92bca03b4bbe: Deleting consensus metadata
I20260929 18:37:57.327193 21788 ts_tablet_manager.cc:1939] T 136a10fa159f42bda9a7d8eb19b811c4 P 1004ea961b344f06b6a5033e2c069359: Deleting consensus metadata
I20260929 18:37:57.327854 21238 catalog_manager.cc:4841] TS 4dc6fe52db3c4e3983ca92bca03b4bbe (127.13.100.2:37145): tablet 136a10fa159f42bda9a7d8eb19b811c4 (table test-table [id=03181ac04c81459f937e0a2cadba8b44]) successfully deleted
I20260929 18:37:57.328004 21237 catalog_manager.cc:4841] TS 1004ea961b344f06b6a5033e2c069359 (127.13.100.1:45113): tablet 136a10fa159f42bda9a7d8eb19b811c4 (table test-table [id=03181ac04c81459f937e0a2cadba8b44]) successfully deleted
I20260929 18:37:57.329053 21789 ts_tablet_manager.cc:1905] T 136a10fa159f42bda9a7d8eb19b811c4 P 9741dfa235754052b0b0c2acc05483e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:37:57.331418 21789 ts_tablet_manager.cc:1918] T 136a10fa159f42bda9a7d8eb19b811c4 P 9741dfa235754052b0b0c2acc05483e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:37:57.331503 21789 log.cc:1196] T 136a10fa159f42bda9a7d8eb19b811c4 P 9741dfa235754052b0b0c2acc05483e6: Deleting WAL directory at /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707036041078-13712-0/minicluster-data/ts-2/wal/wals/136a10fa159f42bda9a7d8eb19b811c4
I20260929 18:37:57.331732 21789 ts_tablet_manager.cc:1939] T 136a10fa159f42bda9a7d8eb19b811c4 P 9741dfa235754052b0b0c2acc05483e6: Deleting consensus metadata
I20260929 18:37:57.332374 21250 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59394:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:37:57.332813 21250 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:37:57.333540 21238 catalog_manager.cc:4841] TS 9741dfa235754052b0b0c2acc05483e6 (127.13.100.3:44055): tablet 136a10fa159f42bda9a7d8eb19b811c4 (table test-table [id=03181ac04c81459f937e0a2cadba8b44]) successfully deleted
I20260929 18:37:57.337602 21389 tablet_service.cc:1467] Processing CreateTablet for tablet 11dfd5a2fdea48f484927a88d6a90e02 (DEFAULT_TABLE table=test-table [id=5ddb487e711e41c29e9ddccaca403b2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:37:57.337603 21529 tablet_service.cc:1467] Processing CreateTablet for tablet 11dfd5a2fdea48f484927a88d6a90e02 (DEFAULT_TABLE table=test-table [id=5ddb487e711e41c29e9ddccaca403b2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:37:57.337862 21529 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11dfd5a2fdea48f484927a88d6a90e02. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:37:57.337862 21389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11dfd5a2fdea48f484927a88d6a90e02. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:37:57.340113 21765 tablet_bootstrap.cc:492] T 11dfd5a2fdea48f484927a88d6a90e02 P 1004ea961b344f06b6a5033e2c069359: Bootstrap starting.
I20260929 18:37:57.340150 21764 tablet_bootstrap.cc:492] T 11dfd5a2fdea48f484927a88d6a90e02 P 4dc6fe52db3c4e3983ca92bca03b4bbe: Bootstrap starting.
I20260929 18:37:57.340902 21765 tablet_bootstrap.cc:654] T 11dfd5a2fdea48f484927a88d6a90e02 P 1004ea961b344f06b6a5033e2c069359: Neither blocks nor log segments found. Creating new log.
I20260929 18:37:57.340924 21764 tablet_bootstrap.cc:654] T 11dfd5a2fdea48f484927a88d6a90e02 P 4dc6fe52db3c4e3983ca92bca03b4bbe: Neither blocks nor log segments found. Creating new log.
I20260929 18:37:57.341805 21669 tablet_service.cc:1467] Processing CreateTablet for tablet 11dfd5a2fdea48f484927a88d6a90e02 (DEFAULT_TABLE table=test-table [id=5ddb487e711e41c29e9ddccaca403b2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:37:57.342104 21669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11dfd5a2fdea48f484927a88d6a90e02. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:37:57.342836 21765 tablet_bootstrap.cc:492] T 11dfd5a2fdea48f484927a88d6a90e02 P 1004ea961b344f06b6a5033e2c069359: No bootstrap required, opened a new log
I20260929 18:37:57.342906 21765 ts_tablet_manager.cc:1397] T 11dfd5a2fdea48f484927a88d6a90e02 P 1004ea961b344f06b6a5033e2c069359: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260929 18:37:57.343269 21765 raft_consensus.cc:348] T 11dfd5a2fdea48f484927a88d6a90e02 P 1004ea961b344f06b6a5033e2c069359 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1004ea961b344f06b6a5033e2c069359" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 45113 } } peers { permanent_uuid: "4dc6fe52db3c4e3983ca92bca03b4bbe" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 37145 } } peers { permanent_uuid: "9741dfa235754052b0b0c2acc05483e6" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 44055 } }
I20260929 18:37:57.343372 21765 raft_consensus.cc:374] T 11dfd5a2fdea48f484927a88d6a90e02 P 1004ea961b344f06b6a5033e2c069359 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:37:57.343402 21765 raft_consensus.cc:729] T 11dfd5a2fdea48f484927a88d6a90e02 P 1004ea961b344f06b6a5033e2c069359 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1004ea961b344f06b6a5033e2c069359, State: Initialized, Role: FOLLOWER
I20260929 18:37:57.343526 21765 consensus_queue.cc:260] T 11dfd5a2fdea48f484927a88d6a90e02 P 1004ea961b344f06b6a5033e2c069359 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1004ea961b344f06b6a5033e2c069359" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 45113 } } peers { permanent_uuid: "4dc6fe52db3c4e3983ca92bca03b4bbe" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 37145 } } peers { permanent_uuid: "9741dfa235754052b0b0c2acc05483e6" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 44055 } }
I20260929 18:37:57.343791 21765 ts_tablet_manager.cc:1428] T 11dfd5a2fdea48f484927a88d6a90e02 P 1004ea961b344f06b6a5033e2c069359: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20260929 18:37:57.344429 21763 tablet_bootstrap.cc:492] T 11dfd5a2fdea48f484927a88d6a90e02 P 9741dfa235754052b0b0c2acc05483e6: Bootstrap starting.
I20260929 18:37:57.345187 21763 tablet_bootstrap.cc:654] T 11dfd5a2fdea48f484927a88d6a90e02 P 9741dfa235754052b0b0c2acc05483e6: Neither blocks nor log segments found. Creating new log.
I20260929 18:37:57.346349 21763 tablet_bootstrap.cc:492] T 11dfd5a2fdea48f484927a88d6a90e02 P 9741dfa235754052b0b0c2acc05483e6: No bootstrap required, opened a new log
I20260929 18:37:57.346433 21763 ts_tablet_manager.cc:1397] T 11dfd5a2fdea48f484927a88d6a90e02 P 9741dfa235754052b0b0c2acc05483e6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 18:37:57.346809 21763 raft_consensus.cc:348] T 11dfd5a2fdea48f484927a88d6a90e02 P 9741dfa235754052b0b0c2acc05483e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1004ea961b344f06b6a5033e2c069359" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 45113 } } peers { permanent_uuid: "4dc6fe52db3c4e3983ca92bca03b4bbe" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 37145 } } peers { permanent_uuid: "9741dfa235754052b0b0c2acc05483e6" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 44055 } }
I20260929 18:37:57.346915 21763 raft_consensus.cc:374] T 11dfd5a2fdea48f484927a88d6a90e02 P 9741dfa235754052b0b0c2acc05483e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:37:57.346957 21763 raft_consensus.cc:729] T 11dfd5a2fdea48f484927a88d6a90e02 P 9741dfa235754052b0b0c2acc05483e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9741dfa235754052b0b0c2acc05483e6, State: Initialized, Role: FOLLOWER
I20260929 18:37:57.347079 21763 consensus_queue.cc:260] T 11dfd5a2fdea48f484927a88d6a90e02 P 9741dfa235754052b0b0c2acc05483e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1004ea961b344f06b6a5033e2c069359" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 45113 } } peers { permanent_uuid: "4dc6fe52db3c4e3983ca92bca03b4bbe" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 37145 } } peers { permanent_uuid: "9741dfa235754052b0b0c2acc05483e6" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 44055 } }
I20260929 18:37:57.347386 21763 ts_tablet_manager.cc:1428] T 11dfd5a2fdea48f484927a88d6a90e02 P 9741dfa235754052b0b0c2acc05483e6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 18:37:57.348281 21764 tablet_bootstrap.cc:492] T 11dfd5a2fdea48f484927a88d6a90e02 P 4dc6fe52db3c4e3983ca92bca03b4bbe: No bootstrap required, opened a new log
I20260929 18:37:57.348376 21764 ts_tablet_manager.cc:1397] T 11dfd5a2fdea48f484927a88d6a90e02 P 4dc6fe52db3c4e3983ca92bca03b4bbe: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20260929 18:37:57.348781 21764 raft_consensus.cc:348] T 11dfd5a2fdea48f484927a88d6a90e02 P 4dc6fe52db3c4e3983ca92bca03b4bbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1004ea961b344f06b6a5033e2c069359" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 45113 } } peers { permanent_uuid: "4dc6fe52db3c4e3983ca92bca03b4bbe" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 37145 } } peers { permanent_uuid: "9741dfa235754052b0b0c2acc05483e6" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 44055 } }
I20260929 18:37:57.348917 21764 raft_consensus.cc:374] T 11dfd5a2fdea48f484927a88d6a90e02 P 4dc6fe52db3c4e3983ca92bca03b4bbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:37:57.348954 21764 raft_consensus.cc:729] T 11dfd5a2fdea48f484927a88d6a90e02 P 4dc6fe52db3c4e3983ca92bca03b4bbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4dc6fe52db3c4e3983ca92bca03b4bbe, State: Initialized, Role: FOLLOWER
I20260929 18:37:57.349102 21764 consensus_queue.cc:260] T 11dfd5a2fdea48f484927a88d6a90e02 P 4dc6fe52db3c4e3983ca92bca03b4bbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1004ea961b344f06b6a5033e2c069359" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 45113 } } peers { permanent_uuid: "4dc6fe52db3c4e3983ca92bca03b4bbe" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 37145 } } peers { permanent_uuid: "9741dfa235754052b0b0c2acc05483e6" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 44055 } }
I20260929 18:37:57.349360 21764 ts_tablet_manager.cc:1428] T 11dfd5a2fdea48f484927a88d6a90e02 P 4dc6fe52db3c4e3983ca92bca03b4bbe: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260929 18:37:57.422700 21769 raft_consensus.cc:482] T 11dfd5a2fdea48f484927a88d6a90e02 P 9741dfa235754052b0b0c2acc05483e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:37:57.422835 21769 raft_consensus.cc:504] T 11dfd5a2fdea48f484927a88d6a90e02 P 9741dfa235754052b0b0c2acc05483e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1004ea961b344f06b6a5033e2c069359" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 45113 } } peers { permanent_uuid: "4dc6fe52db3c4e3983ca92bca03b4bbe" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 37145 } } peers { permanent_uuid: "9741dfa235754052b0b0c2acc05483e6" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 44055 } }
I20260929 18:37:57.423323 21769 leader_election.cc:288] T 11dfd5a2fdea48f484927a88d6a90e02 P 9741dfa235754052b0b0c2acc05483e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1004ea961b344f06b6a5033e2c069359 (127.13.100.1:45113), 4dc6fe52db3c4e3983ca92bca03b4bbe (127.13.100.2:37145)
I20260929 18:37:57.423733 21549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11dfd5a2fdea48f484927a88d6a90e02" candidate_uuid: "9741dfa235754052b0b0c2acc05483e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4dc6fe52db3c4e3983ca92bca03b4bbe" is_pre_election: true
I20260929 18:37:57.423750 21409 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11dfd5a2fdea48f484927a88d6a90e02" candidate_uuid: "9741dfa235754052b0b0c2acc05483e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1004ea961b344f06b6a5033e2c069359" is_pre_election: true
I20260929 18:37:57.423892 21549 raft_consensus.cc:2454] T 11dfd5a2fdea48f484927a88d6a90e02 P 4dc6fe52db3c4e3983ca92bca03b4bbe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9741dfa235754052b0b0c2acc05483e6 in term 0.
I20260929 18:37:57.423909 21409 raft_consensus.cc:2454] T 11dfd5a2fdea48f484927a88d6a90e02 P 1004ea961b344f06b6a5033e2c069359 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9741dfa235754052b0b0c2acc05483e6 in term 0.
I20260929 18:37:57.424204 21621 leader_election.cc:302] T 11dfd5a2fdea48f484927a88d6a90e02 P 9741dfa235754052b0b0c2acc05483e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1004ea961b344f06b6a5033e2c069359, 9741dfa235754052b0b0c2acc05483e6; no voters:
I20260929 18:37:57.424364 21769 raft_consensus.cc:2781] T 11dfd5a2fdea48f484927a88d6a90e02 P 9741dfa235754052b0b0c2acc05483e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:37:57.424427 21769 raft_consensus.cc:482] T 11dfd5a2fdea48f484927a88d6a90e02 P 9741dfa235754052b0b0c2acc05483e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:37:57.424463 21769 raft_consensus.cc:3037] T 11dfd5a2fdea48f484927a88d6a90e02 P 9741dfa235754052b0b0c2acc05483e6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:37:57.425408 21769 raft_consensus.cc:504] T 11dfd5a2fdea48f484927a88d6a90e02 P 9741dfa235754052b0b0c2acc05483e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1004ea961b344f06b6a5033e2c069359" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 45113 } } peers { permanent_uuid: "4dc6fe52db3c4e3983ca92bca03b4bbe" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 37145 } } peers { permanent_uuid: "9741dfa235754052b0b0c2acc05483e6" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 44055 } }
I20260929 18:37:57.425779 21769 leader_election.cc:288] T 11dfd5a2fdea48f484927a88d6a90e02 P 9741dfa235754052b0b0c2acc05483e6 [CANDIDATE]: Term 1 election: Requested vote from peers 1004ea961b344f06b6a5033e2c069359 (127.13.100.1:45113), 4dc6fe52db3c4e3983ca92bca03b4bbe (127.13.100.2:37145)
I20260929 18:37:57.426082 21409 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11dfd5a2fdea48f484927a88d6a90e02" candidate_uuid: "9741dfa235754052b0b0c2acc05483e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1004ea961b344f06b6a5033e2c069359"
I20260929 18:37:57.426153 21549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11dfd5a2fdea48f484927a88d6a90e02" candidate_uuid: "9741dfa235754052b0b0c2acc05483e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4dc6fe52db3c4e3983ca92bca03b4bbe"
I20260929 18:37:57.426226 21409 raft_consensus.cc:3037] T 11dfd5a2fdea48f484927a88d6a90e02 P 1004ea961b344f06b6a5033e2c069359 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:37:57.426278 21549 raft_consensus.cc:3037] T 11dfd5a2fdea48f484927a88d6a90e02 P 4dc6fe52db3c4e3983ca92bca03b4bbe [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:37:57.427223 21409 raft_consensus.cc:2454] T 11dfd5a2fdea48f484927a88d6a90e02 P 1004ea961b344f06b6a5033e2c069359 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9741dfa235754052b0b0c2acc05483e6 in term 1.
I20260929 18:37:57.427491 21621 leader_election.cc:302] T 11dfd5a2fdea48f484927a88d6a90e02 P 9741dfa235754052b0b0c2acc05483e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1004ea961b344f06b6a5033e2c069359, 9741dfa235754052b0b0c2acc05483e6; no voters:
I20260929 18:37:57.427539 21549 raft_consensus.cc:2454] T 11dfd5a2fdea48f484927a88d6a90e02 P 4dc6fe52db3c4e3983ca92bca03b4bbe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9741dfa235754052b0b0c2acc05483e6 in term 1.
I20260929 18:37:57.427683 21769 raft_consensus.cc:2781] T 11dfd5a2fdea48f484927a88d6a90e02 P 9741dfa235754052b0b0c2acc05483e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:37:57.427772 21769 raft_consensus.cc:686] T 11dfd5a2fdea48f484927a88d6a90e02 P 9741dfa235754052b0b0c2acc05483e6 [term 1 LEADER]: Becoming Leader. State: Replica: 9741dfa235754052b0b0c2acc05483e6, State: Running, Role: LEADER
I20260929 18:37:57.427911 21769 consensus_queue.cc:237] T 11dfd5a2fdea48f484927a88d6a90e02 P 9741dfa235754052b0b0c2acc05483e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1004ea961b344f06b6a5033e2c069359" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 45113 } } peers { permanent_uuid: "4dc6fe52db3c4e3983ca92bca03b4bbe" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 37145 } } peers { permanent_uuid: "9741dfa235754052b0b0c2acc05483e6" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 44055 } }
I20260929 18:37:57.429733 21250 catalog_manager.cc:5495] T 11dfd5a2fdea48f484927a88d6a90e02 P 9741dfa235754052b0b0c2acc05483e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9741dfa235754052b0b0c2acc05483e6 (127.13.100.3). New cstate: current_term: 1 leader_uuid: "9741dfa235754052b0b0c2acc05483e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1004ea961b344f06b6a5033e2c069359" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 45113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4dc6fe52db3c4e3983ca92bca03b4bbe" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 37145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9741dfa235754052b0b0c2acc05483e6" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 44055 } health_report { overall_health: HEALTHY } } }
I20260929 18:37:57.462354 21762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:57.458736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48714 (local address 127.13.100.3:44055)
0929 18:37:57.458845 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:57.458848 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:57.458863 (+ 15us) server_negotiation.cc:407] Connection header received
0929 18:37:57.458993 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:57.458996 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:57.459045 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:57.459107 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:37:57.459346 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:57.460007 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:57.461251 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:57.461459 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:57.461550 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:57.461814 (+ 264us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:37:57.461876 (+ 62us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:37:57.462029 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:57.462141 (+ 112us) server_negotiation.cc:299] Negotiation successful
0929 18:37:57.462188 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 18:37:57.468497 21409 raft_consensus.cc:1261] T 11dfd5a2fdea48f484927a88d6a90e02 P 1004ea961b344f06b6a5033e2c069359 [term 1 FOLLOWER]: Refusing update from remote peer 9741dfa235754052b0b0c2acc05483e6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:37:57.468616 21549 raft_consensus.cc:1261] T 11dfd5a2fdea48f484927a88d6a90e02 P 4dc6fe52db3c4e3983ca92bca03b4bbe [term 1 FOLLOWER]: Refusing update from remote peer 9741dfa235754052b0b0c2acc05483e6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:37:57.468943 21795 consensus_queue.cc:1035] T 11dfd5a2fdea48f484927a88d6a90e02 P 9741dfa235754052b0b0c2acc05483e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4dc6fe52db3c4e3983ca92bca03b4bbe" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 37145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:37:57.469098 21769 consensus_queue.cc:1035] T 11dfd5a2fdea48f484927a88d6a90e02 P 9741dfa235754052b0b0c2acc05483e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1004ea961b344f06b6a5033e2c069359" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 45113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:37:57.496168 21250 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59394:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:37:57.496308 21250 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59394:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:37:57.498121 21250 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 73570d97b6164fdb8636f3f3c01b24e4: Sending DeleteTablet for 3 replicas of tablet 11dfd5a2fdea48f484927a88d6a90e02
I20260929 18:37:57.498626 21389 tablet_service.cc:1514] Processing DeleteTablet for tablet 11dfd5a2fdea48f484927a88d6a90e02 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:37:57 UTC) from {username='oryx', principal='oryx/127.13.100.62@KRBTEST.COM'} at 127.0.0.1:39396
I20260929 18:37:57.498925 21669 tablet_service.cc:1514] Processing DeleteTablet for tablet 11dfd5a2fdea48f484927a88d6a90e02 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:37:57 UTC) from {username='oryx', principal='oryx/127.13.100.62@KRBTEST.COM'} at 127.0.0.1:48708
I20260929 18:37:57.498951 21788 tablet_replica.cc:330] T 11dfd5a2fdea48f484927a88d6a90e02 P 1004ea961b344f06b6a5033e2c069359: stopping tablet replica
I20260929 18:37:57.499073 21789 tablet_replica.cc:330] T 11dfd5a2fdea48f484927a88d6a90e02 P 9741dfa235754052b0b0c2acc05483e6: stopping tablet replica
I20260929 18:37:57.499073 21788 raft_consensus.cc:2229] T 11dfd5a2fdea48f484927a88d6a90e02 P 1004ea961b344f06b6a5033e2c069359 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:37:57.499176 21789 raft_consensus.cc:2229] T 11dfd5a2fdea48f484927a88d6a90e02 P 9741dfa235754052b0b0c2acc05483e6 [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:37:57.499193 21788 raft_consensus.cc:2258] T 11dfd5a2fdea48f484927a88d6a90e02 P 1004ea961b344f06b6a5033e2c069359 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:37:57.499296 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 21324
I20260929 18:37:57.499344 21789 raft_consensus.cc:2258] T 11dfd5a2fdea48f484927a88d6a90e02 P 9741dfa235754052b0b0c2acc05483e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:37:57.500524 21529 tablet_service.cc:1514] Processing DeleteTablet for tablet 11dfd5a2fdea48f484927a88d6a90e02 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:37:57 UTC) from {username='oryx', principal='oryx/127.13.100.62@KRBTEST.COM'} at 127.0.0.1:50074
I20260929 18:37:57.500711 21787 tablet_replica.cc:330] T 11dfd5a2fdea48f484927a88d6a90e02 P 4dc6fe52db3c4e3983ca92bca03b4bbe: stopping tablet replica
I20260929 18:37:57.500815 21787 raft_consensus.cc:2229] T 11dfd5a2fdea48f484927a88d6a90e02 P 4dc6fe52db3c4e3983ca92bca03b4bbe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:37:57.500910 21787 raft_consensus.cc:2258] T 11dfd5a2fdea48f484927a88d6a90e02 P 4dc6fe52db3c4e3983ca92bca03b4bbe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:37:57.501657 21787 ts_tablet_manager.cc:1905] T 11dfd5a2fdea48f484927a88d6a90e02 P 4dc6fe52db3c4e3983ca92bca03b4bbe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:37:57.504774 21787 ts_tablet_manager.cc:1918] T 11dfd5a2fdea48f484927a88d6a90e02 P 4dc6fe52db3c4e3983ca92bca03b4bbe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 18:37:57.504860 21787 log.cc:1196] T 11dfd5a2fdea48f484927a88d6a90e02 P 4dc6fe52db3c4e3983ca92bca03b4bbe: Deleting WAL directory at /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707036041078-13712-0/minicluster-data/ts-1/wal/wals/11dfd5a2fdea48f484927a88d6a90e02
I20260929 18:37:57.505129 21787 ts_tablet_manager.cc:1939] T 11dfd5a2fdea48f484927a88d6a90e02 P 4dc6fe52db3c4e3983ca92bca03b4bbe: Deleting consensus metadata
I20260929 18:37:57.505800 21238 catalog_manager.cc:4841] TS 4dc6fe52db3c4e3983ca92bca03b4bbe (127.13.100.2:37145): tablet 11dfd5a2fdea48f484927a88d6a90e02 (table test-table [id=5ddb487e711e41c29e9ddccaca403b2a]) successfully deleted
I20260929 18:37:57.506482 21789 ts_tablet_manager.cc:1905] T 11dfd5a2fdea48f484927a88d6a90e02 P 9741dfa235754052b0b0c2acc05483e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:37:57.508396 21789 ts_tablet_manager.cc:1918] T 11dfd5a2fdea48f484927a88d6a90e02 P 9741dfa235754052b0b0c2acc05483e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 18:37:57.508489 21789 log.cc:1196] T 11dfd5a2fdea48f484927a88d6a90e02 P 9741dfa235754052b0b0c2acc05483e6: Deleting WAL directory at /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707036041078-13712-0/minicluster-data/ts-2/wal/wals/11dfd5a2fdea48f484927a88d6a90e02
I20260929 18:37:57.508773 21789 ts_tablet_manager.cc:1939] T 11dfd5a2fdea48f484927a88d6a90e02 P 9741dfa235754052b0b0c2acc05483e6: Deleting consensus metadata
W20260929 18:37:57.508908 21237 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.100.1:45113 (error 108)
I20260929 18:37:57.509553 21238 catalog_manager.cc:4841] TS 9741dfa235754052b0b0c2acc05483e6 (127.13.100.3:44055): tablet 11dfd5a2fdea48f484927a88d6a90e02 (table test-table [id=5ddb487e711e41c29e9ddccaca403b2a]) successfully deleted
I20260929 18:37:57.510159 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 21464
I20260929 18:37:57.511529 21854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:57.509779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.1:45113 (local address 127.0.0.1:39422)
0929 18:37:57.510108 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:57.510187 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.1:45113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":118,"threads_started":1}
W20260929 18:37:57.511633 21237 catalog_manager.cc:4584] TS 1004ea961b344f06b6a5033e2c069359 (127.13.100.1:45113): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 11dfd5a2fdea48f484927a88d6a90e02: Network error: Client connection negotiation failed: client connection to 127.13.100.1:45113: connect: Connection refused (error 111)
I20260929 18:37:57.518081 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 21604
I20260929 18:37:57.524685 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 21220
2026-09-29T18:37:57Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3336 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790707036041078-13712-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:37:57 dist-test-slave-j1xs krb5kdc[21857](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:37:57 dist-test-slave-j1xs krb5kdc[21857](info): set up 2 sockets
Sep 29 18:37:57 dist-test-slave-j1xs krb5kdc[21857](info): commencing operation
krb5kdc: starting...
W20260929 18:37:58.578474 13712 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790707036041078-13712-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790707036041078-13712-0/krb5kdc/test-user.keytab.
Sep 29 18:37:58 dist-test-slave-j1xs krb5kdc[21857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707078, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:37:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:37:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.100.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.100.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.100.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.62.keytab.
Entry for principal kudu/127.13.100.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.62.keytab.
I20260929 18:37:58.697412 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790707036041078-13712-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790707036041078-13712-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790707036041078-13712-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.100.62:44999
--webserver_interface=127.13.100.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.62.keytab
--principal=kudu/127.13.100.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.100.20:33241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.100.62:44999
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790707036041078-13712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790707036041078-13712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790707036041078-13712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:37:58.805351 21870 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:37:58.805665 21870 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:37:58.805749 21870 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:37:58.809804 21870 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:37:58.809900 21870 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:37:58.809963 21870 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:37:58.809999 21870 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:37:58.810034 21870 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:37:58.815362 21870 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.100.20:33241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790707036041078-13712-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790707036041078-13712-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.100.62:44999
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.100.62:44999
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.100.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21870
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790707036041078-13712-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:37:58.816429 21870 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:37:58.817457 21870 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:37:58.825870 21875 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:37:58.826015 21876 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:37:58.825934 21870 server_base.cc:1034] running on GCE node
W20260929 18:37:58.825871 21878 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:37:58.826772 21870 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:37:58.827782 21870 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:37:58.828958 21870 hybrid_clock.cc:648] HybridClock initialized: now 1790707078828917 us; error 55 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-29T18:37:58Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1309 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:37:58 dist-test-slave-j1xs krb5kdc[21883](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:37:58 dist-test-slave-j1xs krb5kdc[21883](info): set up 2 sockets
Sep 29 18:37:58 dist-test-slave-j1xs krb5kdc[21883](info): commencing operation
krb5kdc: starting...
W20260929 18:37:59.882540 13712 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/krb5kdc/test-user.keytab.
Sep 29 18:37:59 dist-test-slave-j1xs krb5kdc[21883](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707079, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:37:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:37:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.100.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.100.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.100.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.62.keytab.
Entry for principal kudu/127.13.100.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.62.keytab.
I20260929 18:38:00.002816 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.100.62:45525
--webserver_interface=127.13.100.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.62.keytab
--principal=kudu/127.13.100.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.100.20:45187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.100.62:45525
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:38:00.114193 21896 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:38:00.114485 21896 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:38:00.114557 21896 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:38:00.119263 21896 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:38:00.119330 21896 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:38:00.119351 21896 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:38:00.119374 21896 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:38:00.119390 21896 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:38:00.124990 21896 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.100.20:45187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.100.62:45525
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.62.keytab
--principal=kudu/127.13.100.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.100.62:45525
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.100.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21896
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:38:00.126118 21896 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:38:00.127136 21896 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:38:00.135960 21904 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:38:00.135985 21901 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:38:00.135949 21902 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:38:00.136235 21896 server_base.cc:1034] running on GCE node
I20260929 18:38:00.136855 21896 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:38:00.138029 21896 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:38:00.139235 21896 hybrid_clock.cc:648] HybridClock initialized: now 1790707080139213 us; error 56 us; skew 500 ppm
Sep 29 18:38:00 dist-test-slave-j1xs krb5kdc[21883](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707080, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:38:00.142159 21896 init.cc:377] Logged in from keytab as kudu/127.13.100.62@KRBTEST.COM (short username kudu)
I20260929 18:38:00.143108 21896 webserver.cc:460] Webserver started at http://127.13.100.62:40013/ using document root <none> and password file <none>
I20260929 18:38:00.143687 21896 fs_manager.cc:359] Metadata directory not provided
I20260929 18:38:00.143749 21896 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:38:00.143967 21896 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:38:00.146153 21896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/minicluster-data/master-0/data/instance:
uuid: "9b6aad3273174abd938e1644908d7043"
format_stamp: "Formatted at 2026-09-29 18:38:00 on dist-test-slave-j1xs"
server_key: "8da22b55dfe047e0359e456a7ebe7583"
server_key_iv: "4c6b5e7bd436386dd67b227eb1870888"
server_key_version: "encryptionkey@0"
I20260929 18:38:00.146720 21896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/minicluster-data/master-0/wal/instance:
uuid: "9b6aad3273174abd938e1644908d7043"
format_stamp: "Formatted at 2026-09-29 18:38:00 on dist-test-slave-j1xs"
server_key: "8da22b55dfe047e0359e456a7ebe7583"
server_key_iv: "4c6b5e7bd436386dd67b227eb1870888"
server_key_version: "encryptionkey@0"
I20260929 18:38:00.150611 21896 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 18:38:00.152957 21911 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:38:00.154486 21896 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:38:00.154592 21896 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/minicluster-data/master-0/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/minicluster-data/master-0/wal
uuid: "9b6aad3273174abd938e1644908d7043"
format_stamp: "Formatted at 2026-09-29 18:38:00 on dist-test-slave-j1xs"
server_key: "8da22b55dfe047e0359e456a7ebe7583"
server_key_iv: "4c6b5e7bd436386dd67b227eb1870888"
server_key_version: "encryptionkey@0"
I20260929 18:38:00.154734 21896 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:38:00.184820 21896 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:38:00.188299 21896 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:38:00.188514 21896 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:38:00.196240 21896 rpc_server.cc:307] RPC server started. Bound to: 127.13.100.62:45525
I20260929 18:38:00.196267 21963 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.100.62:45525 every 8 connection(s)
I20260929 18:38:00.197304 21896 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
I20260929 18:38:00.199463 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 21896
I20260929 18:38:00.199565 13712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/minicluster-data/master-0/wal/instance
I20260929 18:38:00.199872 13712 external_mini_cluster.cc:1428] Setting key a788017ff5ca6dca1fb46f4054945fa9
I20260929 18:38:00.200188 21964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:38:00.206373 21964 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b6aad3273174abd938e1644908d7043: Bootstrap starting.
I20260929 18:38:00.208484 21964 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9b6aad3273174abd938e1644908d7043: Neither blocks nor log segments found. Creating new log.
Sep 29 18:38:00 dist-test-slave-j1xs krb5kdc[21883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707079, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.100.62@KRBTEST.COM
I20260929 18:38:00.209263 21964 log.cc:824] T 00000000000000000000000000000000 P 9b6aad3273174abd938e1644908d7043: Log is configured to *not* fsync() on all Append() calls
I20260929 18:38:00.211127 21964 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b6aad3273174abd938e1644908d7043: No bootstrap required, opened a new log
I20260929 18:38:00.214576 21964 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9b6aad3273174abd938e1644908d7043 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b6aad3273174abd938e1644908d7043" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 45525 } }
I20260929 18:38:00.214752 21967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:00.201289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54594 (local address 127.13.100.62:45525)
0929 18:38:00.201718 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0929 18:38:00.201732 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 18:38:00.201767 (+ 35us) server_negotiation.cc:407] Connection header received
0929 18:38:00.202878 (+ 1111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:38:00.202897 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:38:00.203218 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:38:00.203569 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:38:00.204381 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:00.205381 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:00.206294 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:00.206810 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:00.206977 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:00.209731 (+ 2754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:38:00.209736 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:38:00.209750 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:38:00.209779 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:38:00.211757 (+ 1978us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:38:00.212230 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:38:00.212244 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:38:00.212248 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:38:00.212369 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:38:00.212675 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:38:00.212679 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:38:00.212681 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:38:00.213087 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:38:00.213314 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0929 18:38:00.213607 (+ 293us) server_negotiation.cc:299] Negotiation successful
0929 18:38:00.213976 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":293,"thread_start_us":132,"threads_started":1}
I20260929 18:38:00.214843 21964 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9b6aad3273174abd938e1644908d7043 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:38:00.214907 21964 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9b6aad3273174abd938e1644908d7043 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b6aad3273174abd938e1644908d7043, State: Initialized, Role: FOLLOWER
I20260929 18:38:00.215404 21964 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9b6aad3273174abd938e1644908d7043 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b6aad3273174abd938e1644908d7043" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 45525 } }
I20260929 18:38:00.215543 21964 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9b6aad3273174abd938e1644908d7043 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:38:00.215602 21964 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9b6aad3273174abd938e1644908d7043 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:38:00.215777 21964 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9b6aad3273174abd938e1644908d7043 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:38:00.216708 21964 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9b6aad3273174abd938e1644908d7043 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b6aad3273174abd938e1644908d7043" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 45525 } }
I20260929 18:38:00.217037 21964 leader_election.cc:302] T 00000000000000000000000000000000 P 9b6aad3273174abd938e1644908d7043 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9b6aad3273174abd938e1644908d7043; no voters:
I20260929 18:38:00.217391 21964 leader_election.cc:288] T 00000000000000000000000000000000 P 9b6aad3273174abd938e1644908d7043 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:38:00.217509 21969 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9b6aad3273174abd938e1644908d7043 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:38:00.217703 21969 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9b6aad3273174abd938e1644908d7043 [term 1 LEADER]: Becoming Leader. State: Replica: 9b6aad3273174abd938e1644908d7043, State: Running, Role: LEADER
I20260929 18:38:00.218075 21969 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9b6aad3273174abd938e1644908d7043 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b6aad3273174abd938e1644908d7043" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 45525 } }
I20260929 18:38:00.218569 21964 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9b6aad3273174abd938e1644908d7043 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:38:00.219689 21970 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b6aad3273174abd938e1644908d7043 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9b6aad3273174abd938e1644908d7043" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b6aad3273174abd938e1644908d7043" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 45525 } } }
I20260929 18:38:00.219831 21970 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b6aad3273174abd938e1644908d7043 [sys.catalog]: This master's current role is: LEADER
I20260929 18:38:00.220144 21971 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b6aad3273174abd938e1644908d7043 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9b6aad3273174abd938e1644908d7043. Latest consensus state: current_term: 1 leader_uuid: "9b6aad3273174abd938e1644908d7043" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b6aad3273174abd938e1644908d7043" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 45525 } } }
I20260929 18:38:00.220233 21971 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b6aad3273174abd938e1644908d7043 [sys.catalog]: This master's current role is: LEADER
I20260929 18:38:00.220310 21978 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:38:00.222978 21978 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:38:00.228245 21978 catalog_manager.cc:1347] Generated new cluster ID: 02c2fd0dc2fa46f49a163ca1636c3c38
I20260929 18:38:00.228315 21978 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:38:00.246430 21978 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:38:00.247727 21978 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:38:00.256834 21978 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9b6aad3273174abd938e1644908d7043: Generated new TSK 0
I20260929 18:38:00.257669 21978 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.100.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.100.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.100.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.1.keytab.
Entry for principal kudu/127.13.100.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.1.keytab.
I20260929 18:38:00.293171 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.100.1:0
--local_ip_for_outbound_sockets=127.13.100.1
--webserver_interface=127.13.100.1
--webserver_port=0
--tserver_master_addrs=127.13.100.62:45525
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.1.keytab
--principal=kudu/127.13.100.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.100.20:45187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:38:00.404338 21990 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:38:00.404584 21990 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:38:00.404626 21990 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:38:00.408646 21990 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:38:00.408706 21990 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:38:00.408763 21990 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.100.1
I20260929 18:38:00.414384 21990 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.100.20:45187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.1.keytab
--principal=kudu/127.13.100.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.100.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.100.1
--webserver_port=0
--tserver_master_addrs=127.13.100.62:45525
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21990
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.100.1
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:38:00.415369 21990 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:38:00.416313 21990 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:38:00.425734 21995 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:38:00.425791 21996 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:38:00.425966 21990 server_base.cc:1034] running on GCE node
W20260929 18:38:00.425789 21998 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:38:00.426422 21990 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:38:00.427022 21990 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:38:00.428215 21990 hybrid_clock.cc:648] HybridClock initialized: now 1790707080428191 us; error 33 us; skew 500 ppm
Sep 29 18:38:00 dist-test-slave-j1xs krb5kdc[21883](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707080, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:38:00.431211 21990 init.cc:377] Logged in from keytab as kudu/127.13.100.1@KRBTEST.COM (short username kudu)
I20260929 18:38:00.432045 21990 webserver.cc:460] Webserver started at http://127.13.100.1:35813/ using document root <none> and password file <none>
I20260929 18:38:00.432587 21990 fs_manager.cc:359] Metadata directory not provided
I20260929 18:38:00.432647 21990 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:38:00.432852 21990 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:38:00.434845 21990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/minicluster-data/ts-0/data/instance:
uuid: "3007d23be33e446689e294daa0e450f3"
format_stamp: "Formatted at 2026-09-29 18:38:00 on dist-test-slave-j1xs"
server_key: "edfdb98cad8b76191c94fdf718d39b77"
server_key_iv: "8060935677ad76056088d0e81ff0899c"
server_key_version: "encryptionkey@0"
I20260929 18:38:00.435379 21990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/minicluster-data/ts-0/wal/instance:
uuid: "3007d23be33e446689e294daa0e450f3"
format_stamp: "Formatted at 2026-09-29 18:38:00 on dist-test-slave-j1xs"
server_key: "edfdb98cad8b76191c94fdf718d39b77"
server_key_iv: "8060935677ad76056088d0e81ff0899c"
server_key_version: "encryptionkey@0"
I20260929 18:38:00.438975 21990 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 18:38:00.441607 22005 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:38:00.442832 21990 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:38:00.442955 21990 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/minicluster-data/ts-0/wal
uuid: "3007d23be33e446689e294daa0e450f3"
format_stamp: "Formatted at 2026-09-29 18:38:00 on dist-test-slave-j1xs"
server_key: "edfdb98cad8b76191c94fdf718d39b77"
server_key_iv: "8060935677ad76056088d0e81ff0899c"
server_key_version: "encryptionkey@0"
I20260929 18:38:00.443053 21990 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:38:00.462491 21990 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:38:00.465479 21990 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:38:00.465673 21990 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:38:00.466297 21990 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:38:00.467094 21990 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:38:00.467139 21990 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:38:00.467214 21990 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:38:00.467247 21990 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:38:00.475911 21990 rpc_server.cc:307] RPC server started. Bound to: 127.13.100.1:35551
I20260929 18:38:00.475931 22118 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.100.1:35551 every 8 connection(s)
I20260929 18:38:00.476775 21990 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/minicluster-data/ts-0/data/info.pb
I20260929 18:38:00.479503 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 21990
I20260929 18:38:00.479637 13712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/minicluster-data/ts-0/wal/instance
I20260929 18:38:00.479904 13712 external_mini_cluster.cc:1428] Setting key c7d793a687a15c3336bed7dd32f9b15d
Sep 29 18:38:00 dist-test-slave-j1xs krb5kdc[21883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707080, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.1@KRBTEST.COM for kudu/127.13.100.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.13.100.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.100.2@KRBTEST.COM" created.
I20260929 18:38:00.496227 21967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:00.478455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.1:56461 (local address 127.13.100.62:45525)
0929 18:38:00.478594 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0929 18:38:00.478599 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:38:00.479438 (+ 839us) server_negotiation.cc:407] Connection header received
0929 18:38:00.480842 (+ 1404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:38:00.480845 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:38:00.480896 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:38:00.481000 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:38:00.482184 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:00.482709 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:00.483826 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:00.484062 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:00.484174 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:00.486981 (+ 2807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:38:00.486984 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:38:00.486986 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:38:00.487008 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:38:00.493808 (+ 6800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:38:00.494382 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:38:00.494389 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:38:00.494393 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:38:00.494494 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:38:00.494921 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:38:00.494927 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:38:00.494930 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:38:00.495190 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:38:00.495310 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0929 18:38:00.495808 (+ 498us) server_negotiation.cc:299] Negotiation successful
0929 18:38:00.495962 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 18:38:00.496862 22121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:00.478881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:45525 (local address 127.13.100.1:56461)
0929 18:38:00.479296 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:00.479337 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 18:38:00.480632 (+ 1295us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:38:00.481164 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:38:00.481170 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:38:00.481509 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:38:00.482045 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:00.482061 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:00.482842 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:00.482845 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:00.483690 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:00.483699 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:00.484211 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:00.484215 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:00.484384 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:00.484991 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:38:00.485006 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:38:00.486815 (+ 1809us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:38:00.493987 (+ 7172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:38:00.493993 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:38:00.494005 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:38:00.494271 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:38:00.494601 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:38:00.494603 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:38:00.494604 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:38:00.494807 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:38:00.495309 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:38:00.495314 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:38:00.495548 (+ 234us) client_negotiation.cc:769] Sending connection context
0929 18:38:00.495721 (+ 173us) client_negotiation.cc:240] Negotiation successful
0929 18:38:00.495975 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":112,"threads_started":1}
I20260929 18:38:00.498131 22119 heartbeater.cc:347] Connected to a master server at 127.13.100.62:45525
I20260929 18:38:00.498356 22119 heartbeater.cc:464] Registering TS with master...
I20260929 18:38:00.498842 22119 heartbeater.cc:511] Master 127.13.100.62:45525 requested a full tablet report, sending...
I20260929 18:38:00.500358 21928 ts_manager.cc:194] Registered new tserver with Master: 3007d23be33e446689e294daa0e450f3 (127.13.100.1:35551)
I20260929 18:38:00.501447 21928 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.100.1@KRBTEST.COM'} at 127.13.100.1:56461
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.100.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.2.keytab.
Entry for principal kudu/127.13.100.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.2.keytab.
I20260929 18:38:00.507316 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.100.2:0
--local_ip_for_outbound_sockets=127.13.100.2
--webserver_interface=127.13.100.2
--webserver_port=0
--tserver_master_addrs=127.13.100.62:45525
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.2.keytab
--principal=kudu/127.13.100.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.100.20:45187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:38:00.619357 22124 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:38:00.619678 22124 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:38:00.619726 22124 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:38:00.623914 22124 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:38:00.623973 22124 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:38:00.624030 22124 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.100.2
I20260929 18:38:00.629112 22124 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.100.20:45187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.2.keytab
--principal=kudu/127.13.100.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.100.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.100.2
--webserver_port=0
--tserver_master_addrs=127.13.100.62:45525
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22124
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.100.2
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:38:00.630201 22124 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:38:00.631151 22124 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 18:38:00.640936 22124 server_base.cc:1034] running on GCE node
W20260929 18:38:00.640908 22129 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:38:00.640832 22132 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:38:00.640837 22130 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:38:00.641525 22124 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:38:00.642190 22124 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:38:00.643404 22124 hybrid_clock.cc:648] HybridClock initialized: now 1790707080643389 us; error 29 us; skew 500 ppm
Sep 29 18:38:00 dist-test-slave-j1xs krb5kdc[21883](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707080, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:38:00.646466 22124 init.cc:377] Logged in from keytab as kudu/127.13.100.2@KRBTEST.COM (short username kudu)
I20260929 18:38:00.647389 22124 webserver.cc:460] Webserver started at http://127.13.100.2:39529/ using document root <none> and password file <none>
I20260929 18:38:00.647965 22124 fs_manager.cc:359] Metadata directory not provided
I20260929 18:38:00.648026 22124 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:38:00.648243 22124 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:38:00.650367 22124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/minicluster-data/ts-1/data/instance:
uuid: "2d438ae72c0d4a21a7e4248e139b4ce5"
format_stamp: "Formatted at 2026-09-29 18:38:00 on dist-test-slave-j1xs"
server_key: "afe989768089034f184d8d5e4b9ac2c0"
server_key_iv: "1bb303cacf4cbbc38f7acaa91feb2b4d"
server_key_version: "encryptionkey@0"
I20260929 18:38:00.650921 22124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/minicluster-data/ts-1/wal/instance:
uuid: "2d438ae72c0d4a21a7e4248e139b4ce5"
format_stamp: "Formatted at 2026-09-29 18:38:00 on dist-test-slave-j1xs"
server_key: "afe989768089034f184d8d5e4b9ac2c0"
server_key_iv: "1bb303cacf4cbbc38f7acaa91feb2b4d"
server_key_version: "encryptionkey@0"
I20260929 18:38:00.654702 22124 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 18:38:00.657167 22139 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:38:00.658613 22124 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:38:00.658768 22124 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/minicluster-data/ts-1/wal
uuid: "2d438ae72c0d4a21a7e4248e139b4ce5"
format_stamp: "Formatted at 2026-09-29 18:38:00 on dist-test-slave-j1xs"
server_key: "afe989768089034f184d8d5e4b9ac2c0"
server_key_iv: "1bb303cacf4cbbc38f7acaa91feb2b4d"
server_key_version: "encryptionkey@0"
I20260929 18:38:00.658876 22124 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:38:00.685139 22124 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:38:00.688234 22124 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:38:00.688441 22124 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:38:00.689033 22124 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:38:00.689867 22124 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:38:00.689936 22124 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:38:00.690004 22124 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:38:00.690043 22124 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:38:00.699219 22124 rpc_server.cc:307] RPC server started. Bound to: 127.13.100.2:33097
I20260929 18:38:00.699244 22252 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.100.2:33097 every 8 connection(s)
I20260929 18:38:00.700111 22124 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/minicluster-data/ts-1/data/info.pb
I20260929 18:38:00.703779 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 22124
I20260929 18:38:00.703886 13712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/minicluster-data/ts-1/wal/instance
I20260929 18:38:00.704238 13712 external_mini_cluster.cc:1428] Setting key 85c3a35caaa329653267a77461b0e8ea
Sep 29 18:38:00 dist-test-slave-j1xs krb5kdc[21883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707080, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.2@KRBTEST.COM for kudu/127.13.100.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.13.100.3@KRBTEST.COM; defaulting to no policy
I20260929 18:38:00.714011 21967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:00.701993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.2:56017 (local address 127.13.100.62:45525)
0929 18:38:00.702145 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0929 18:38:00.702151 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:38:00.702999 (+ 848us) server_negotiation.cc:407] Connection header received
0929 18:38:00.704441 (+ 1442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:38:00.704444 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:38:00.704502 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:38:00.704591 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:38:00.705753 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:00.706349 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:00.707489 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:00.707731 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:00.707843 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:00.710465 (+ 2622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:38:00.710475 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:38:00.710478 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:38:00.710503 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:38:00.711877 (+ 1374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:38:00.712389 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:38:00.712394 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:38:00.712396 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:38:00.712476 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:38:00.712810 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:38:00.712815 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:38:00.712817 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:38:00.713044 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:38:00.713151 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 18:38:00.713686 (+ 535us) server_negotiation.cc:299] Negotiation successful
0929 18:38:00.713811 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 18:38:00.715027 22255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:00.702408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:45525 (local address 127.13.100.2:56017)
0929 18:38:00.702852 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:00.702893 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 18:38:00.704196 (+ 1303us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:38:00.704771 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:38:00.704777 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:38:00.705137 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:38:00.705604 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:00.705618 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:00.706478 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:00.706481 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:00.707360 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:00.707369 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:00.707878 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:00.707882 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:00.708056 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:00.708674 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:38:00.708690 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:38:00.710304 (+ 1614us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:38:00.711996 (+ 1692us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:38:00.712002 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:38:00.712016 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:38:00.712287 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:38:00.712566 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:38:00.712571 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:38:00.712573 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:38:00.712712 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:38:00.713146 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:38:00.713153 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:38:00.713407 (+ 254us) client_negotiation.cc:769] Sending connection context
0929 18:38:00.713589 (+ 182us) client_negotiation.cc:240] Negotiation successful
0929 18:38:00.713941 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":113,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.100.3@KRBTEST.COM" created.
I20260929 18:38:00.716241 22253 heartbeater.cc:347] Connected to a master server at 127.13.100.62:45525
I20260929 18:38:00.716480 22253 heartbeater.cc:464] Registering TS with master...
I20260929 18:38:00.716948 22253 heartbeater.cc:511] Master 127.13.100.62:45525 requested a full tablet report, sending...
I20260929 18:38:00.718000 21928 ts_manager.cc:194] Registered new tserver with Master: 2d438ae72c0d4a21a7e4248e139b4ce5 (127.13.100.2:33097)
I20260929 18:38:00.718876 21928 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.100.2@KRBTEST.COM'} at 127.13.100.2:56017
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.100.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.3.keytab.
Entry for principal kudu/127.13.100.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.3.keytab.
I20260929 18:38:00.730541 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.100.3:0
--local_ip_for_outbound_sockets=127.13.100.3
--webserver_interface=127.13.100.3
--webserver_port=0
--tserver_master_addrs=127.13.100.62:45525
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.3.keytab
--principal=kudu/127.13.100.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.100.20:45187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:38:00.842883 22258 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:38:00.843137 22258 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:38:00.843182 22258 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:38:00.847201 22258 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:38:00.847278 22258 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:38:00.847354 22258 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.100.3
I20260929 18:38:00.852516 22258 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.100.20:45187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.3.keytab
--principal=kudu/127.13.100.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.100.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.100.3
--webserver_port=0
--tserver_master_addrs=127.13.100.62:45525
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22258
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.100.3
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:38:00.853431 22258 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:38:00.854432 22258 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:38:00.864002 22264 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:38:00.863981 22266 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:38:00.864187 22258 server_base.cc:1034] running on GCE node
W20260929 18:38:00.864085 22263 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:38:00.864699 22258 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:38:00.865363 22258 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:38:00.866545 22258 hybrid_clock.cc:648] HybridClock initialized: now 1790707080866534 us; error 27 us; skew 500 ppm
Sep 29 18:38:00 dist-test-slave-j1xs krb5kdc[21883](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707080, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:38:00.869288 22258 init.cc:377] Logged in from keytab as kudu/127.13.100.3@KRBTEST.COM (short username kudu)
I20260929 18:38:00.870203 22258 webserver.cc:460] Webserver started at http://127.13.100.3:46817/ using document root <none> and password file <none>
I20260929 18:38:00.870738 22258 fs_manager.cc:359] Metadata directory not provided
I20260929 18:38:00.870824 22258 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:38:00.871054 22258 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:38:00.872982 22258 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/minicluster-data/ts-2/data/instance:
uuid: "d1d65f6b1e634cc1a775b8ecbc7652b4"
format_stamp: "Formatted at 2026-09-29 18:38:00 on dist-test-slave-j1xs"
server_key: "5535787941f286f7b30e28f59e88287b"
server_key_iv: "2f952dfbe4ddf6830b927e3dc0cb933e"
server_key_version: "encryptionkey@0"
I20260929 18:38:00.873514 22258 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/minicluster-data/ts-2/wal/instance:
uuid: "d1d65f6b1e634cc1a775b8ecbc7652b4"
format_stamp: "Formatted at 2026-09-29 18:38:00 on dist-test-slave-j1xs"
server_key: "5535787941f286f7b30e28f59e88287b"
server_key_iv: "2f952dfbe4ddf6830b927e3dc0cb933e"
server_key_version: "encryptionkey@0"
I20260929 18:38:00.877058 22258 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260929 18:38:00.879447 22273 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:38:00.880664 22258 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 18:38:00.880793 22258 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/minicluster-data/ts-2/wal
uuid: "d1d65f6b1e634cc1a775b8ecbc7652b4"
format_stamp: "Formatted at 2026-09-29 18:38:00 on dist-test-slave-j1xs"
server_key: "5535787941f286f7b30e28f59e88287b"
server_key_iv: "2f952dfbe4ddf6830b927e3dc0cb933e"
server_key_version: "encryptionkey@0"
I20260929 18:38:00.880898 22258 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:38:00.904147 22258 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:38:00.906980 22258 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:38:00.907189 22258 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:38:00.907768 22258 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:38:00.908557 22258 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:38:00.908628 22258 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:38:00.908697 22258 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:38:00.908749 22258 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:38:00.917342 22258 rpc_server.cc:307] RPC server started. Bound to: 127.13.100.3:45375
I20260929 18:38:00.917369 22386 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.100.3:45375 every 8 connection(s)
I20260929 18:38:00.918270 22258 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/minicluster-data/ts-2/data/info.pb
I20260929 18:38:00.926997 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 22258
I20260929 18:38:00.927132 13712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/minicluster-data/ts-2/wal/instance
I20260929 18:38:00.927445 13712 external_mini_cluster.cc:1428] Setting key 7f1f52536bd8acdd992402dfb4a20251
Sep 29 18:38:00 dist-test-slave-j1xs krb5kdc[21883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707080, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.3@KRBTEST.COM for kudu/127.13.100.62@KRBTEST.COM
I20260929 18:38:00.931846 21967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:00.920050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.3:36503 (local address 127.13.100.62:45525)
0929 18:38:00.920163 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0929 18:38:00.920166 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:38:00.920990 (+ 824us) server_negotiation.cc:407] Connection header received
0929 18:38:00.922312 (+ 1322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:38:00.922315 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:38:00.922367 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:38:00.922445 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:38:00.923504 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:00.924035 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:00.925137 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:00.925374 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:00.925487 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:00.928141 (+ 2654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:38:00.928145 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:38:00.928147 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:38:00.928175 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:38:00.929812 (+ 1637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:38:00.930474 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:38:00.930477 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:38:00.930478 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:38:00.930568 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:38:00.930899 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:38:00.930902 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:38:00.930904 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:38:00.931074 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:38:00.931150 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 18:38:00.931638 (+ 488us) server_negotiation.cc:299] Negotiation successful
0929 18:38:00.931734 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260929 18:38:00.932683 22389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:00.920447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:45525 (local address 127.13.100.3:36503)
0929 18:38:00.920852 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:00.920891 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 18:38:00.922113 (+ 1222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:38:00.922594 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:38:00.922600 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:38:00.922921 (+ 321us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:38:00.923363 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:00.923375 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:00.924159 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:00.924163 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:00.925004 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:00.925012 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:00.925523 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:00.925527 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:00.925691 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:00.926258 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:38:00.926273 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:38:00.927948 (+ 1675us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:38:00.929991 (+ 2043us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:38:00.929997 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:38:00.930010 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:38:00.930366 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:38:00.930656 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:38:00.930658 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:38:00.930659 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:38:00.930790 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:38:00.931191 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:38:00.931197 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:38:00.931432 (+ 235us) client_negotiation.cc:769] Sending connection context
0929 18:38:00.931581 (+ 149us) client_negotiation.cc:240] Negotiation successful
0929 18:38:00.931813 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":110,"threads_started":1}
I20260929 18:38:00.933737 22387 heartbeater.cc:347] Connected to a master server at 127.13.100.62:45525
I20260929 18:38:00.934005 22387 heartbeater.cc:464] Registering TS with master...
I20260929 18:38:00.934430 22387 heartbeater.cc:511] Master 127.13.100.62:45525 requested a full tablet report, sending...
I20260929 18:38:00.935395 21928 ts_manager.cc:194] Registered new tserver with Master: d1d65f6b1e634cc1a775b8ecbc7652b4 (127.13.100.3:45375)
I20260929 18:38:00.935943 21928 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.100.3@KRBTEST.COM'} at 127.13.100.3:36503
I20260929 18:38:00.941289 13712 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.13.100.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.100.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.100.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/krb5kdc/oryx_127.13.100.1.keytab.
Entry for principal oryx/127.13.100.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/krb5kdc/oryx_127.13.100.1.keytab.
WARNING: no policy specified for oryx/127.13.100.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.100.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.100.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/krb5kdc/oryx_127.13.100.2.keytab.
Entry for principal oryx/127.13.100.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/krb5kdc/oryx_127.13.100.2.keytab.
WARNING: no policy specified for oryx/127.13.100.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.100.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.100.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/krb5kdc/oryx_127.13.100.3.keytab.
Entry for principal oryx/127.13.100.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/krb5kdc/oryx_127.13.100.3.keytab.
I20260929 18:38:01.010735 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 21990
I20260929 18:38:01.016248 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 22124
I20260929 18:38:01.021662 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 22258
I20260929 18:38:01.027124 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 21896
I20260929 18:38:01.033232 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.100.62:45525
--webserver_interface=127.13.100.62
--webserver_port=40013
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.62.keytab
--principal=kudu/127.13.100.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.100.20:45187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.100.62:45525
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:38:01.152148 22396 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:38:01.152431 22396 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:38:01.152496 22396 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:38:01.156967 22396 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:38:01.157049 22396 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:38:01.157073 22396 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:38:01.157094 22396 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:38:01.157150 22396 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:38:01.163033 22396 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.100.20:45187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.100.62:45525
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.62.keytab
--principal=kudu/127.13.100.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.100.62:45525
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.100.62
--webserver_port=40013
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22396
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:38:01.164305 22396 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:38:01.165364 22396 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:38:01.173938 22402 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:38:01.173965 22401 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:38:01.173967 22404 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:38:01.174402 22396 server_base.cc:1034] running on GCE node
I20260929 18:38:01.175007 22396 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:38:01.176193 22396 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:38:01.177368 22396 hybrid_clock.cc:648] HybridClock initialized: now 1790707081177352 us; error 33 us; skew 500 ppm
Sep 29 18:38:01 dist-test-slave-j1xs krb5kdc[21883](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707081, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:38:01.180367 22396 init.cc:377] Logged in from keytab as kudu/127.13.100.62@KRBTEST.COM (short username kudu)
I20260929 18:38:01.181303 22396 webserver.cc:460] Webserver started at http://127.13.100.62:40013/ using document root <none> and password file <none>
I20260929 18:38:01.181861 22396 fs_manager.cc:359] Metadata directory not provided
I20260929 18:38:01.182035 22396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:38:01.185781 22396 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:38:01.187928 22411 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:38:01.189172 22396 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:38:01.189311 22396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/minicluster-data/master-0/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/minicluster-data/master-0/wal
uuid: "9b6aad3273174abd938e1644908d7043"
format_stamp: "Formatted at 2026-09-29 18:38:00 on dist-test-slave-j1xs"
server_key: "8da22b55dfe047e0359e456a7ebe7583"
server_key_iv: "4c6b5e7bd436386dd67b227eb1870888"
server_key_version: "encryptionkey@0"
I20260929 18:38:01.189836 22396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:38:01.202602 22396 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:38:01.205626 22396 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:38:01.205832 22396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:38:01.212483 22396 rpc_server.cc:307] RPC server started. Bound to: 127.13.100.62:45525
I20260929 18:38:01.212508 22463 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.100.62:45525 every 8 connection(s)
I20260929 18:38:01.213434 22396 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
I20260929 18:38:01.217378 22464 sys_catalog.cc:263] Verifying existing consensus state
I20260929 18:38:01.218930 22464 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b6aad3273174abd938e1644908d7043: Bootstrap starting.
I20260929 18:38:01.219761 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 22396
I20260929 18:38:01.220273 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.100.1:35551
--local_ip_for_outbound_sockets=127.13.100.1
--tserver_master_addrs=127.13.100.62:45525
--webserver_port=35813
--webserver_interface=127.13.100.1
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.1.keytab
--principal=kudu/127.13.100.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.100.20:45187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.13.100.1
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/krb5kdc/oryx_127.13.100.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 18:38:01.235287 22464 log.cc:824] T 00000000000000000000000000000000 P 9b6aad3273174abd938e1644908d7043: Log is configured to *not* fsync() on all Append() calls
I20260929 18:38:01.241528 22464 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b6aad3273174abd938e1644908d7043: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 18:38:01.241880 22464 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b6aad3273174abd938e1644908d7043: Bootstrap complete.
I20260929 18:38:01.246060 22464 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9b6aad3273174abd938e1644908d7043 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b6aad3273174abd938e1644908d7043" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 45525 } }
I20260929 18:38:01.246475 22464 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9b6aad3273174abd938e1644908d7043 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b6aad3273174abd938e1644908d7043, State: Initialized, Role: FOLLOWER
I20260929 18:38:01.246938 22464 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9b6aad3273174abd938e1644908d7043 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b6aad3273174abd938e1644908d7043" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 45525 } }
I20260929 18:38:01.247089 22464 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9b6aad3273174abd938e1644908d7043 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:38:01.247169 22464 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9b6aad3273174abd938e1644908d7043 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:38:01.247285 22464 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9b6aad3273174abd938e1644908d7043 [term 1 FOLLOWER]: Advancing to term 2
I20260929 18:38:01.249110 22464 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9b6aad3273174abd938e1644908d7043 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b6aad3273174abd938e1644908d7043" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 45525 } }
I20260929 18:38:01.249436 22464 leader_election.cc:302] T 00000000000000000000000000000000 P 9b6aad3273174abd938e1644908d7043 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9b6aad3273174abd938e1644908d7043; no voters:
I20260929 18:38:01.249752 22464 leader_election.cc:288] T 00000000000000000000000000000000 P 9b6aad3273174abd938e1644908d7043 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 18:38:01.249948 22468 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9b6aad3273174abd938e1644908d7043 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 18:38:01.250281 22468 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9b6aad3273174abd938e1644908d7043 [term 2 LEADER]: Becoming Leader. State: Replica: 9b6aad3273174abd938e1644908d7043, State: Running, Role: LEADER
I20260929 18:38:01.250702 22468 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9b6aad3273174abd938e1644908d7043 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b6aad3273174abd938e1644908d7043" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 45525 } }
I20260929 18:38:01.250957 22464 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9b6aad3273174abd938e1644908d7043 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:38:01.252120 22469 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b6aad3273174abd938e1644908d7043 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "9b6aad3273174abd938e1644908d7043" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b6aad3273174abd938e1644908d7043" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 45525 } } }
I20260929 18:38:01.252254 22469 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b6aad3273174abd938e1644908d7043 [sys.catalog]: This master's current role is: LEADER
I20260929 18:38:01.252657 22470 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b6aad3273174abd938e1644908d7043 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9b6aad3273174abd938e1644908d7043. Latest consensus state: current_term: 2 leader_uuid: "9b6aad3273174abd938e1644908d7043" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b6aad3273174abd938e1644908d7043" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 45525 } } }
I20260929 18:38:01.252803 22470 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b6aad3273174abd938e1644908d7043 [sys.catalog]: This master's current role is: LEADER
I20260929 18:38:01.253186 22478 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:38:01.256428 22478 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:38:01.257114 22478 catalog_manager.cc:1259] Loaded cluster ID: 02c2fd0dc2fa46f49a163ca1636c3c38
I20260929 18:38:01.257179 22478 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:38:01.259562 22478 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:38:01.260257 22478 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9b6aad3273174abd938e1644908d7043: Loaded TSK: 0
I20260929 18:38:01.261437 22478 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 18:38:01.334615 22466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:38:01.334869 22466 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:38:01.334941 22466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:38:01.338960 22466 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:38:01.339021 22466 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:38:01.339124 22466 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.100.1
I20260929 18:38:01.344228 22466 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.100.20:45187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/krb5kdc/oryx_127.13.100.1.keytab
--principal=oryx/127.13.100.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.100.1:35551
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.100.1
--webserver_port=35813
--tserver_master_addrs=127.13.100.62:45525
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22466
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.100.1
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:38:01.345237 22466 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:38:01.346277 22466 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:38:01.355870 22491 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:38:01.355885 22490 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:38:01.355876 22493 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:38:01.356364 22466 server_base.cc:1034] running on GCE node
I20260929 18:38:01.356838 22466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:38:01.357555 22466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:38:01.358772 22466 hybrid_clock.cc:648] HybridClock initialized: now 1790707081358748 us; error 35 us; skew 500 ppm
Sep 29 18:38:01 dist-test-slave-j1xs krb5kdc[21883](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707081, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.100.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:38:01.362107 22466 init.cc:377] Logged in from keytab as oryx/127.13.100.1@KRBTEST.COM (short username oryx)
I20260929 18:38:01.363022 22466 webserver.cc:460] Webserver started at http://127.13.100.1:35813/ using document root <none> and password file <none>
I20260929 18:38:01.363659 22466 fs_manager.cc:359] Metadata directory not provided
I20260929 18:38:01.363744 22466 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:38:01.367753 22466 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260929 18:38:01.369972 22500 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:38:01.371223 22466 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260929 18:38:01.371364 22466 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/minicluster-data/ts-0/wal
uuid: "3007d23be33e446689e294daa0e450f3"
format_stamp: "Formatted at 2026-09-29 18:38:00 on dist-test-slave-j1xs"
server_key: "edfdb98cad8b76191c94fdf718d39b77"
server_key_iv: "8060935677ad76056088d0e81ff0899c"
server_key_version: "encryptionkey@0"
I20260929 18:38:01.371871 22466 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:38:01.400861 22466 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:38:01.404316 22466 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:38:01.404531 22466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:38:01.405143 22466 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:38:01.405988 22466 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:38:01.406068 22466 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:38:01.406140 22466 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:38:01.406193 22466 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:38:01.415091 22466 rpc_server.cc:307] RPC server started. Bound to: 127.13.100.1:35551
I20260929 18:38:01.415112 22613 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.100.1:35551 every 8 connection(s)
I20260929 18:38:01.416079 22466 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/minicluster-data/ts-0/data/info.pb
I20260929 18:38:01.416692 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 22466
I20260929 18:38:01.417132 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.100.2:33097
--local_ip_for_outbound_sockets=127.13.100.2
--tserver_master_addrs=127.13.100.62:45525
--webserver_port=39529
--webserver_interface=127.13.100.2
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.2.keytab
--principal=kudu/127.13.100.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.100.20:45187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.13.100.2
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/krb5kdc/oryx_127.13.100.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 18:38:01 dist-test-slave-j1xs krb5kdc[21883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.100.1@KRBTEST.COM for oryx/127.13.100.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:38:01 dist-test-slave-j1xs krb5kdc[21883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.100.1@KRBTEST.COM for oryx/127.13.100.62@KRBTEST.COM, Server not found in Kerberos database
W20260929 18:38:01.432082 22617 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:38:01.418337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.1:41431 (local address 127.13.100.62:45525)
0929 18:38:01.418852 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0929 18:38:01.418874 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0929 18:38:01.419167 (+ 293us) server_negotiation.cc:407] Connection header received
0929 18:38:01.421225 (+ 2058us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:38:01.421238 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:38:01.421670 (+ 432us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:38:01.422121 (+ 451us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:38:01.423813 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:01.424780 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:01.426682 (+ 1902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:01.427133 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:01.427261 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:01.431375 (+ 4114us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.100.1:41431: BlockingRecv error: recv got EOF from 127.13.100.1:41431 (error 108)
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":141,"threads_started":1}
I20260929 18:38:01.432102 22618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:01.418497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:45525 (local address 127.13.100.1:41431)
0929 18:38:01.418994 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:01.419044 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 18:38:01.420780 (+ 1736us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:38:01.422292 (+ 1512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:38:01.422303 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:38:01.422861 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:38:01.423610 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:01.423633 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:01.424928 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:01.424934 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:01.426553 (+ 1619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:01.426567 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:01.427309 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:01.427315 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:01.427545 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:01.428223 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:38:01.428249 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:38:01.431154 (+ 2905us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.100.62:45525: Server oryx/127.13.100.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":124,"threads_started":1}
W20260929 18:38:01.432771 22614 heartbeater.cc:644] Failed to heartbeat to 127.13.100.62:45525 (0 consecutive failures): Not authorized: Failed to ping master at 127.13.100.62:45525: Client connection negotiation failed: client connection to 127.13.100.62:45525: Server oryx/127.13.100.62@KRBTEST.COM not found in Kerberos database
Sep 29 18:38:01 dist-test-slave-j1xs krb5kdc[21883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.100.1@KRBTEST.COM for oryx/127.13.100.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:38:01 dist-test-slave-j1xs krb5kdc[21883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.100.1@KRBTEST.COM for oryx/127.13.100.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:38:01.439080 22618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:01.433227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:45525 (local address 127.13.100.1:48303)
0929 18:38:01.433392 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:01.433409 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 18:38:01.433490 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:38:01.433710 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:38:01.433711 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:38:01.433849 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:38:01.434000 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:01.434006 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:01.435012 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:01.435014 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:01.435590 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:01.435598 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:01.436208 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:01.436211 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:01.436284 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:01.436902 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:38:01.436915 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:38:01.438983 (+ 2068us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.100.62:45525: Server oryx/127.13.100.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":85}
W20260929 18:38:01.439374 22617 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:38:01.433346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.1:48303 (local address 127.13.100.62:45525)
0929 18:38:01.433459 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0929 18:38:01.433463 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:38:01.433477 (+ 14us) server_negotiation.cc:407] Connection header received
0929 18:38:01.433569 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:38:01.433572 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:38:01.433615 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:38:01.433688 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:38:01.434094 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:01.434905 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:01.435723 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:01.436076 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:01.436178 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:01.439141 (+ 2963us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.100.1:48303: BlockingRecv error: recv got EOF from 127.13.100.1:48303 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 29 18:38:01 dist-test-slave-j1xs krb5kdc[21883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.100.1@KRBTEST.COM for oryx/127.13.100.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:38:01 dist-test-slave-j1xs krb5kdc[21883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.100.1@KRBTEST.COM for oryx/127.13.100.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:38:01.444783 22618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:01.439666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:45525 (local address 127.13.100.1:52203)
0929 18:38:01.439755 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:01.439771 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 18:38:01.439846 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:38:01.440060 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:38:01.440062 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:38:01.440184 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:38:01.440248 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:01.440253 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:01.441190 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:01.441193 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:01.441825 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:01.441835 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:01.442409 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:01.442411 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:01.442469 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:01.442978 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:38:01.442990 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:38:01.444688 (+ 1698us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.100.62:45525: Server oryx/127.13.100.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":27}
W20260929 18:38:01.444931 22617 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:38:01.439739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.1:52203 (local address 127.13.100.62:45525)
0929 18:38:01.439841 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0929 18:38:01.439845 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:38:01.439859 (+ 14us) server_negotiation.cc:407] Connection header received
0929 18:38:01.439918 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:38:01.439920 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:38:01.439964 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:38:01.440035 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:38:01.440347 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:01.441060 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:01.442003 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:01.442298 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:01.442397 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:01.444833 (+ 2436us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.100.1:52203: BlockingRecv error: recv got EOF from 127.13.100.1:52203 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260929 18:38:01.445065 22614 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 18:38:01.536412 22616 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:38:01.536687 22616 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:38:01.536736 22616 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:38:01.540894 22616 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:38:01.540969 22616 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:38:01.541031 22616 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.100.2
I20260929 18:38:01.546509 22616 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.100.20:45187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/krb5kdc/oryx_127.13.100.2.keytab
--principal=oryx/127.13.100.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.100.2:33097
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.100.2
--webserver_port=39529
--tserver_master_addrs=127.13.100.62:45525
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22616
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.100.2
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:38:01.547629 22616 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:38:01.548658 22616 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 18:38:01.560127 22616 server_base.cc:1034] running on GCE node
W20260929 18:38:01.560343 22626 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:38:01.560353 22624 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:38:01.560416 22623 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:38:01.561056 22616 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:38:01.561748 22616 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:38:01.562960 22616 hybrid_clock.cc:648] HybridClock initialized: now 1790707081562945 us; error 36 us; skew 500 ppm
Sep 29 18:38:01 dist-test-slave-j1xs krb5kdc[21883](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707081, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.100.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:38:01.566113 22616 init.cc:377] Logged in from keytab as oryx/127.13.100.2@KRBTEST.COM (short username oryx)
I20260929 18:38:01.567217 22616 webserver.cc:460] Webserver started at http://127.13.100.2:39529/ using document root <none> and password file <none>
I20260929 18:38:01.567838 22616 fs_manager.cc:359] Metadata directory not provided
I20260929 18:38:01.567956 22616 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:38:01.574326 22616 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 18:38:01.577414 22633 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:38:01.579044 22616 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:38:01.579223 22616 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/minicluster-data/ts-1/wal
uuid: "2d438ae72c0d4a21a7e4248e139b4ce5"
format_stamp: "Formatted at 2026-09-29 18:38:00 on dist-test-slave-j1xs"
server_key: "afe989768089034f184d8d5e4b9ac2c0"
server_key_iv: "1bb303cacf4cbbc38f7acaa91feb2b4d"
server_key_version: "encryptionkey@0"
I20260929 18:38:01.579814 22616 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:38:01.604533 22616 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:38:01.608027 22616 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:38:01.608239 22616 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:38:01.608862 22616 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:38:01.609704 22616 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:38:01.609777 22616 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:38:01.609855 22616 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:38:01.609923 22616 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:38:01.620390 22616 rpc_server.cc:307] RPC server started. Bound to: 127.13.100.2:33097
I20260929 18:38:01.620410 22746 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.100.2:33097 every 8 connection(s)
I20260929 18:38:01.621420 22616 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/minicluster-data/ts-1/data/info.pb
I20260929 18:38:01.625662 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 22616
I20260929 18:38:01.626228 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.100.3:45375
--local_ip_for_outbound_sockets=127.13.100.3
--tserver_master_addrs=127.13.100.62:45525
--webserver_port=46817
--webserver_interface=127.13.100.3
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.3.keytab
--principal=kudu/127.13.100.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.100.20:45187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.13.100.3
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/krb5kdc/oryx_127.13.100.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 18:38:01 dist-test-slave-j1xs krb5kdc[21883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.100.2@KRBTEST.COM for oryx/127.13.100.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:38:01 dist-test-slave-j1xs krb5kdc[21883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.100.2@KRBTEST.COM for oryx/127.13.100.62@KRBTEST.COM, Server not found in Kerberos database
W20260929 18:38:01.633960 22617 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:38:01.623309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.2:59953 (local address 127.13.100.62:45525)
0929 18:38:01.623484 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0929 18:38:01.623490 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:38:01.624309 (+ 819us) server_negotiation.cc:407] Connection header received
0929 18:38:01.625749 (+ 1440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:38:01.625755 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:38:01.625820 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:38:01.625944 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:38:01.627508 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:01.628029 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:01.629545 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:01.630000 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:01.630132 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:01.633758 (+ 3626us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.100.2:59953: BlockingRecv error: recv got EOF from 127.13.100.2:59953 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 18:38:01.634429 22749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:01.623711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:45525 (local address 127.13.100.2:59953)
0929 18:38:01.624141 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:01.624182 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 18:38:01.625451 (+ 1269us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:38:01.626122 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:38:01.626133 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:38:01.626660 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:38:01.627345 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:01.627365 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:01.628158 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:01.628163 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:01.629409 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:01.629418 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:01.630145 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:01.630148 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:01.630307 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:01.630998 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:38:01.631015 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:38:01.633556 (+ 2541us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.100.62:45525: Server oryx/127.13.100.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":110,"threads_started":1}
W20260929 18:38:01.635020 22747 heartbeater.cc:644] Failed to heartbeat to 127.13.100.62:45525 (0 consecutive failures): Not authorized: Failed to ping master at 127.13.100.62:45525: Client connection negotiation failed: client connection to 127.13.100.62:45525: Server oryx/127.13.100.62@KRBTEST.COM not found in Kerberos database
Sep 29 18:38:01 dist-test-slave-j1xs krb5kdc[21883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.100.2@KRBTEST.COM for oryx/127.13.100.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:38:01 dist-test-slave-j1xs krb5kdc[21883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.100.2@KRBTEST.COM for oryx/127.13.100.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:38:01.641203 22749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:01.635467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:45525 (local address 127.13.100.2:40719)
0929 18:38:01.635624 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:01.635642 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 18:38:01.635738 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:38:01.635986 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:38:01.635992 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:38:01.636183 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:38:01.636316 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:01.636325 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:01.637096 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:01.637103 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:01.638112 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:01.638123 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:01.638582 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:01.638586 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:01.638675 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:01.639151 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:38:01.639165 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:38:01.641075 (+ 1910us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.100.62:45525: Server oryx/127.13.100.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":65}
W20260929 18:38:01.641317 22617 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:38:01.635579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.2:40719 (local address 127.13.100.62:45525)
0929 18:38:01.635678 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0929 18:38:01.635682 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:38:01.635694 (+ 12us) server_negotiation.cc:407] Connection header received
0929 18:38:01.635835 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:38:01.635837 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:38:01.635871 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:38:01.635935 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:38:01.636499 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:01.636985 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:01.638224 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:01.638473 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:01.638559 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:01.641190 (+ 2631us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.100.2:40719: BlockingRecv error: recv got EOF from 127.13.100.2:40719 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
Sep 29 18:38:01 dist-test-slave-j1xs krb5kdc[21883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.100.2@KRBTEST.COM for oryx/127.13.100.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:38:01 dist-test-slave-j1xs krb5kdc[21883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.100.2@KRBTEST.COM for oryx/127.13.100.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:38:01.646831 22749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:01.641706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:45525 (local address 127.13.100.2:59451)
0929 18:38:01.641882 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:01.641895 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:38:01.641997 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:38:01.642208 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:38:01.642210 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:38:01.642342 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:38:01.642406 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:01.642410 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:01.643389 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:01.643392 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:01.643896 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:01.643903 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:01.644496 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:01.644499 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:01.644554 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:01.645031 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:38:01.645042 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:38:01.646734 (+ 1692us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.100.62:45525: Server oryx/127.13.100.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113}
W20260929 18:38:01.647022 22747 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 18:38:01.647152 22617 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:38:01.641821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.2:59451 (local address 127.13.100.62:45525)
0929 18:38:01.641955 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0929 18:38:01.641959 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:38:01.641972 (+ 13us) server_negotiation.cc:407] Connection header received
0929 18:38:01.642074 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:38:01.642077 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:38:01.642116 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:38:01.642185 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:38:01.642503 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:01.643271 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:01.644008 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:01.644385 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:01.644490 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:01.646871 (+ 2381us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.100.2:59451: BlockingRecv error: recv got EOF from 127.13.100.2:59451 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260929 18:38:01.741780 22750 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:38:01.742048 22750 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:38:01.742089 22750 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:38:01.746114 22750 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:38:01.746176 22750 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:38:01.746277 22750 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.100.3
I20260929 18:38:01.751659 22750 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.100.20:45187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/krb5kdc/oryx_127.13.100.3.keytab
--principal=oryx/127.13.100.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.100.3:45375
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.100.3
--webserver_port=46817
--tserver_master_addrs=127.13.100.62:45525
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22750
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.100.3
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:38:01.752912 22750 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:38:01.754026 22750 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:38:01.763342 22758 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:38:01.763334 22750 server_base.cc:1034] running on GCE node
W20260929 18:38:01.763326 22755 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:38:01.763396 22756 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:38:01.763974 22750 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:38:01.764703 22750 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:38:01.765977 22750 hybrid_clock.cc:648] HybridClock initialized: now 1790707081765956 us; error 32 us; skew 500 ppm
Sep 29 18:38:01 dist-test-slave-j1xs krb5kdc[21883](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707081, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.100.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:38:01.768776 22750 init.cc:377] Logged in from keytab as oryx/127.13.100.3@KRBTEST.COM (short username oryx)
I20260929 18:38:01.770092 22750 webserver.cc:460] Webserver started at http://127.13.100.3:46817/ using document root <none> and password file <none>
I20260929 18:38:01.770637 22750 fs_manager.cc:359] Metadata directory not provided
I20260929 18:38:01.770828 22750 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:38:01.774863 22750 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 18:38:01.777110 22765 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:38:01.778664 22750 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 18:38:01.778762 22750 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/minicluster-data/ts-2/wal
uuid: "d1d65f6b1e634cc1a775b8ecbc7652b4"
format_stamp: "Formatted at 2026-09-29 18:38:00 on dist-test-slave-j1xs"
server_key: "5535787941f286f7b30e28f59e88287b"
server_key_iv: "2f952dfbe4ddf6830b927e3dc0cb933e"
server_key_version: "encryptionkey@0"
I20260929 18:38:01.779271 22750 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:38:01.791891 22750 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:38:01.795354 22750 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:38:01.795501 22750 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:38:01.796012 22750 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:38:01.796756 22750 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:38:01.796799 22750 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:38:01.796838 22750 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:38:01.796854 22750 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:38:01.806196 22750 rpc_server.cc:307] RPC server started. Bound to: 127.13.100.3:45375
I20260929 18:38:01.806221 22878 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.100.3:45375 every 8 connection(s)
I20260929 18:38:01.807210 22750 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707036041078-13712-0/minicluster-data/ts-2/data/info.pb
I20260929 18:38:01.813364 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 22750
Sep 29 18:38:01 dist-test-slave-j1xs krb5kdc[21883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.100.3@KRBTEST.COM for oryx/127.13.100.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:38:01 dist-test-slave-j1xs krb5kdc[21883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.100.3@KRBTEST.COM for oryx/127.13.100.62@KRBTEST.COM, Server not found in Kerberos database
W20260929 18:38:01.819448 22617 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:38:01.809193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.3:45321 (local address 127.13.100.62:45525)
0929 18:38:01.809348 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0929 18:38:01.809352 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:38:01.810789 (+ 1437us) server_negotiation.cc:407] Connection header received
0929 18:38:01.811649 (+ 860us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:38:01.811651 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:38:01.811699 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:38:01.811786 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:38:01.812941 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:01.813548 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:01.815044 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:01.815247 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:01.815322 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:01.819288 (+ 3966us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.100.3:45321: BlockingRecv error: recv got EOF from 127.13.100.3:45321 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 18:38:01.820046 22881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:01.809614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:45525 (local address 127.13.100.3:45321)
0929 18:38:01.810621 (+ 1007us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:01.810670 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 18:38:01.811460 (+ 790us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:38:01.811958 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:38:01.811964 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:38:01.812347 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:38:01.812810 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:01.812822 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:01.813679 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:01.813683 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:01.814930 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:01.814941 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:01.815406 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:01.815412 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:01.815651 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:01.816346 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:38:01.816364 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:38:01.819099 (+ 2735us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.100.62:45525: Server oryx/127.13.100.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":732,"thread_start_us":579,"threads_started":1}
W20260929 18:38:01.820637 22879 heartbeater.cc:644] Failed to heartbeat to 127.13.100.62:45525 (0 consecutive failures): Not authorized: Failed to ping master at 127.13.100.62:45525: Client connection negotiation failed: client connection to 127.13.100.62:45525: Server oryx/127.13.100.62@KRBTEST.COM not found in Kerberos database
I20260929 18:38:01.824410 22883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:01.814529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43700 (local address 127.13.100.62:45525)
0929 18:38:01.814757 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0929 18:38:01.814760 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:38:01.814770 (+ 10us) server_negotiation.cc:407] Connection header received
0929 18:38:01.814797 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:38:01.814799 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:38:01.814833 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:38:01.814903 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:38:01.815899 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:01.816377 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:01.817499 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:01.817775 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:01.817852 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:01.819611 (+ 1759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:38:01.819618 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:38:01.819636 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:38:01.819670 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:38:01.822078 (+ 2408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:38:01.822506 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:38:01.822513 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:38:01.822519 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:38:01.822684 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:38:01.823114 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:38:01.823117 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:38:01.823118 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:38:01.823522 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:38:01.823713 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0929 18:38:01.824041 (+ 328us) server_negotiation.cc:299] Negotiation successful
0929 18:38:01.824287 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":96,"threads_started":1}
Sep 29 18:38:01 dist-test-slave-j1xs krb5kdc[21883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.100.3@KRBTEST.COM for oryx/127.13.100.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:38:01 dist-test-slave-j1xs krb5kdc[21883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.100.3@KRBTEST.COM for oryx/127.13.100.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:38:01.827314 22881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:01.821175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:45525 (local address 127.13.100.3:58567)
0929 18:38:01.821306 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:01.821320 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:38:01.821406 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:38:01.821654 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:38:01.821657 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:38:01.821801 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:38:01.821959 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:01.821967 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:01.822875 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:01.822879 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:01.823600 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:01.823614 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:01.824150 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:01.824157 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:01.824297 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:01.824777 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:38:01.824789 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:38:01.827174 (+ 2385us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.100.62:45525: Server oryx/127.13.100.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":48}
W20260929 18:38:01.827730 22617 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:38:01.821253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.3:58567 (local address 127.13.100.62:45525)
0929 18:38:01.821396 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0929 18:38:01.821402 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:38:01.821420 (+ 18us) server_negotiation.cc:407] Connection header received
0929 18:38:01.821468 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:38:01.821471 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:38:01.821513 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:38:01.821611 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:38:01.822086 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:01.822749 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:01.823737 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:01.824012 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:01.824114 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:01.827620 (+ 3506us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.100.3:58567: BlockingRecv error: recv got EOF from 127.13.100.3:58567 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 29 18:38:01 dist-test-slave-j1xs krb5kdc[21883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.100.3@KRBTEST.COM for oryx/127.13.100.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:38:01 dist-test-slave-j1xs krb5kdc[21883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.100.3@KRBTEST.COM for oryx/127.13.100.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:38:01.832743 22881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:01.827906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:45525 (local address 127.13.100.3:36987)
0929 18:38:01.828028 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:01.828040 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:38:01.828100 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:38:01.828429 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:38:01.828433 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:38:01.828582 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:38:01.828647 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:01.828652 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:01.829351 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:01.829357 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:01.829931 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:01.829940 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:01.830378 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:01.830381 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:01.830456 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:01.830906 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:38:01.830922 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:38:01.832608 (+ 1686us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.100.62:45525: Server oryx/127.13.100.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":49}
W20260929 18:38:01.832816 22617 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:38:01.828112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.3:36987 (local address 127.13.100.62:45525)
0929 18:38:01.828253 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0929 18:38:01.828256 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:38:01.828265 (+ 9us) server_negotiation.cc:407] Connection header received
0929 18:38:01.828298 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:38:01.828300 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:38:01.828331 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:38:01.828415 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:38:01.828755 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:01.829238 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:01.830052 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:01.830263 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:01.830378 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:01.832709 (+ 2331us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.100.3:36987: BlockingRecv error: recv got EOF from 127.13.100.3:36987 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260929 18:38:01.833005 22879 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 29 18:38:02 dist-test-slave-j1xs krb5kdc[21883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.100.1@KRBTEST.COM for oryx/127.13.100.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:38:02 dist-test-slave-j1xs krb5kdc[21883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.100.1@KRBTEST.COM for oryx/127.13.100.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:38:02.450881 22884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:02.445630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:45525 (local address 127.13.100.1:57555)
0929 18:38:02.445906 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:02.445924 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 18:38:02.446032 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:38:02.446230 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:38:02.446232 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:38:02.446411 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:38:02.446513 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:02.446519 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:02.447261 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:02.447264 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:02.447849 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:02.447857 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:02.448282 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:02.448285 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:02.448349 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:02.448865 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:38:02.448885 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:38:02.450718 (+ 1833us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.100.62:45525: Server oryx/127.13.100.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":63,"threads_started":1}
W20260929 18:38:02.451041 22885 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:38:02.445765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.1:57555 (local address 127.13.100.62:45525)
0929 18:38:02.445996 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0929 18:38:02.445999 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:38:02.446011 (+ 12us) server_negotiation.cc:407] Connection header received
0929 18:38:02.446110 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:38:02.446112 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:38:02.446153 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:38:02.446220 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:38:02.446621 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:02.447145 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:02.447960 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:02.448174 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:02.448253 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:02.450849 (+ 2596us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.100.1:57555: BlockingRecv error: recv got EOF from 127.13.100.1:57555 (error 108)
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":69,"threads_started":1}
Sep 29 18:38:02 dist-test-slave-j1xs krb5kdc[21883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.100.2@KRBTEST.COM for oryx/127.13.100.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:38:02 dist-test-slave-j1xs krb5kdc[21883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.100.2@KRBTEST.COM for oryx/127.13.100.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:38:02.653853 22886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:02.647698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:45525 (local address 127.13.100.2:55415)
0929 18:38:02.648083 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:02.648098 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 18:38:02.648231 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:38:02.648503 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:38:02.648506 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:38:02.648723 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:38:02.648815 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:02.648820 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:02.649932 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:02.649936 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:02.650777 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:02.650787 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:02.651218 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:02.651221 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:02.651285 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:02.651889 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:38:02.651902 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:38:02.653735 (+ 1833us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.100.62:45525: Server oryx/127.13.100.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":103,"threads_started":1}
W20260929 18:38:02.654042 22885 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:38:02.647867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.2:55415 (local address 127.13.100.62:45525)
0929 18:38:02.648043 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0929 18:38:02.648047 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:38:02.648181 (+ 134us) server_negotiation.cc:407] Connection header received
0929 18:38:02.648335 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:38:02.648347 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:38:02.648398 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:38:02.648483 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:38:02.648933 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:02.649675 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:02.650893 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:02.651112 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:02.651188 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:02.653874 (+ 2686us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.100.2:55415: BlockingRecv error: recv got EOF from 127.13.100.2:55415 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Sep 29 18:38:02 dist-test-slave-j1xs krb5kdc[21883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.100.3@KRBTEST.COM for oryx/127.13.100.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:38:02 dist-test-slave-j1xs krb5kdc[21883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.100.3@KRBTEST.COM for oryx/127.13.100.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:38:02.839102 22887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:02.833665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:45525 (local address 127.13.100.3:34537)
0929 18:38:02.833990 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:02.834003 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:38:02.834139 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:38:02.834411 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:38:02.834414 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:38:02.834616 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:38:02.834702 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:02.834707 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:02.835464 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:02.835471 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:02.836123 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:02.836131 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:02.836580 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:02.836583 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:02.836666 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:02.837156 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:38:02.837168 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:38:02.838931 (+ 1763us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.100.62:45525: Server oryx/127.13.100.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":102,"threads_started":1}
W20260929 18:38:02.839156 22885 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:38:02.833733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.3:34537 (local address 127.13.100.62:45525)
0929 18:38:02.833910 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0929 18:38:02.833916 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:38:02.834081 (+ 165us) server_negotiation.cc:407] Connection header received
0929 18:38:02.834239 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:38:02.834243 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:38:02.834301 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:38:02.834390 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:38:02.834819 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:02.835357 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:02.836239 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:02.836471 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:02.836577 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:02.839047 (+ 2470us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.100.3:34537: BlockingRecv error: recv got EOF from 127.13.100.3:34537 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 29 18:38:03 dist-test-slave-j1xs krb5kdc[21883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.100.1@KRBTEST.COM for oryx/127.13.100.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:38:03 dist-test-slave-j1xs krb5kdc[21883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.100.1@KRBTEST.COM for oryx/127.13.100.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:38:03.457464 22888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:03.451720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:45525 (local address 127.13.100.1:49443)
0929 18:38:03.452027 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:03.452043 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 18:38:03.452160 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:38:03.452549 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:38:03.452553 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:38:03.452820 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:38:03.452926 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:03.452932 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:03.453672 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:03.453675 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:03.454270 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:03.454279 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:03.454691 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:03.454693 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:03.454760 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:03.455292 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:38:03.455304 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:38:03.457297 (+ 1993us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.100.62:45525: Server oryx/127.13.100.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":106,"threads_started":1}
W20260929 18:38:03.457607 22889 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:38:03.451762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.1:49443 (local address 127.13.100.62:45525)
0929 18:38:03.452092 (+ 330us) server_negotiation.cc:206] Beginning negotiation
0929 18:38:03.452095 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:38:03.452107 (+ 12us) server_negotiation.cc:407] Connection header received
0929 18:38:03.452291 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:38:03.452294 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:38:03.452349 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:38:03.452438 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:38:03.453033 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:03.453558 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:03.454373 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:03.454588 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:03.454665 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:03.457454 (+ 2789us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.100.1:49443: BlockingRecv error: recv got EOF from 127.13.100.1:49443 (error 108)
Metrics: {"server-negotiator.queue_time_us":247,"thread_start_us":79,"threads_started":1}
Sep 29 18:38:03 dist-test-slave-j1xs krb5kdc[21883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.100.2@KRBTEST.COM for oryx/127.13.100.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:38:03 dist-test-slave-j1xs krb5kdc[21883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.100.2@KRBTEST.COM for oryx/127.13.100.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:38:03.660286 22890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:03.654720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:45525 (local address 127.13.100.2:48983)
0929 18:38:03.655002 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:03.655017 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 18:38:03.655156 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:38:03.655437 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:38:03.655440 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:38:03.655646 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:38:03.655746 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:03.655751 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:03.656538 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:03.656544 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:03.657098 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:03.657106 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:03.657552 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:03.657555 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:03.657641 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:03.658258 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:38:03.658283 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:38:03.660088 (+ 1805us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.100.62:45525: Server oryx/127.13.100.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":84,"threads_started":1}
W20260929 18:38:03.660307 22889 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:38:03.654782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.2:48983 (local address 127.13.100.62:45525)
0929 18:38:03.654972 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0929 18:38:03.654976 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:38:03.655103 (+ 127us) server_negotiation.cc:407] Connection header received
0929 18:38:03.655271 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:38:03.655274 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:38:03.655328 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:38:03.655418 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:38:03.655871 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:03.656428 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:03.657222 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:03.657440 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:03.657548 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:03.660210 (+ 2662us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.100.2:48983: BlockingRecv error: recv got EOF from 127.13.100.2:48983 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
Sep 29 18:38:03 dist-test-slave-j1xs krb5kdc[21883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.100.3@KRBTEST.COM for oryx/127.13.100.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:38:03 dist-test-slave-j1xs krb5kdc[21883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.100.3@KRBTEST.COM for oryx/127.13.100.62@KRBTEST.COM, Server not found in Kerberos database
W20260929 18:38:03.845187 22889 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:38:03.840025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.3:36493 (local address 127.13.100.62:45525)
0929 18:38:03.840157 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 18:38:03.840161 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:38:03.840275 (+ 114us) server_negotiation.cc:407] Connection header received
0929 18:38:03.840380 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:38:03.840382 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:38:03.840425 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:38:03.840491 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:38:03.840972 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:03.841511 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:03.842296 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:03.842508 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:03.842582 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:03.845093 (+ 2511us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.100.3:36493: BlockingRecv error: recv got EOF from 127.13.100.3:36493 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 18:38:03.845198 22891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:03.839875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:45525 (local address 127.13.100.3:36493)
0929 18:38:03.840195 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:03.840208 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:38:03.840304 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:38:03.840583 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:38:03.840586 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:38:03.840762 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:38:03.840864 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:03.840870 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:03.841607 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:03.841609 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:03.842205 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:03.842213 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:03.842601 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:03.842604 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:03.842660 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:03.843246 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:38:03.843257 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:38:03.845048 (+ 1791us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.100.62:45525: Server oryx/127.13.100.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":117,"threads_started":1}
Sep 29 18:38:04 dist-test-slave-j1xs krb5kdc[21883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.100.1@KRBTEST.COM for oryx/127.13.100.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:38:04 dist-test-slave-j1xs krb5kdc[21883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.100.1@KRBTEST.COM for oryx/127.13.100.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:38:04.463863 22892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:04.458269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:45525 (local address 127.13.100.1:58341)
0929 18:38:04.458566 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:04.458579 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:38:04.458667 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:38:04.458903 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:38:04.458905 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:38:04.459079 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:38:04.459172 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:04.459177 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:04.459947 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:04.459950 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:04.460498 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:04.460506 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:04.461000 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:04.461005 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:04.461088 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:04.461610 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:38:04.461622 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:38:04.463753 (+ 2131us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.100.62:45525: Server oryx/127.13.100.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":87,"threads_started":1}
W20260929 18:38:04.464013 22893 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:38:04.458401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.1:58341 (local address 127.13.100.62:45525)
0929 18:38:04.458625 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0929 18:38:04.458628 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:38:04.458642 (+ 14us) server_negotiation.cc:407] Connection header received
0929 18:38:04.458750 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:38:04.458752 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:38:04.458793 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:38:04.458875 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:38:04.459298 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:04.459839 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:04.460630 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:04.460864 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:04.460957 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:04.463884 (+ 2927us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.100.1:58341: BlockingRecv error: recv got EOF from 127.13.100.1:58341 (error 108)
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":85,"threads_started":1}
Sep 29 18:38:04 dist-test-slave-j1xs krb5kdc[21883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.100.2@KRBTEST.COM for oryx/127.13.100.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:38:04 dist-test-slave-j1xs krb5kdc[21883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.100.2@KRBTEST.COM for oryx/127.13.100.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:38:04.666402 22894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:04.661196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:45525 (local address 127.13.100.2:39181)
0929 18:38:04.661457 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:04.661479 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 18:38:04.661593 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:38:04.661821 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:38:04.661823 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:38:04.662024 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:38:04.662123 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:04.662129 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:04.662939 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:04.662943 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:04.663506 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:04.663514 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:04.663934 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:04.663936 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:04.663994 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:04.664478 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:38:04.664490 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:38:04.666288 (+ 1798us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.100.62:45525: Server oryx/127.13.100.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":86,"threads_started":1}
W20260929 18:38:04.666554 22893 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:38:04.661262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.2:39181 (local address 127.13.100.62:45525)
0929 18:38:04.661379 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0929 18:38:04.661382 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:38:04.661611 (+ 229us) server_negotiation.cc:407] Connection header received
0929 18:38:04.661671 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:38:04.661673 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:38:04.661718 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:38:04.661783 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:38:04.662226 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:04.662808 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:04.663626 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:04.663835 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:04.663919 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:04.666427 (+ 2508us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.100.2:39181: BlockingRecv error: recv got EOF from 127.13.100.2:39181 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Sep 29 18:38:04 dist-test-slave-j1xs krb5kdc[21883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.100.3@KRBTEST.COM for oryx/127.13.100.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:38:04 dist-test-slave-j1xs krb5kdc[21883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.100.3@KRBTEST.COM for oryx/127.13.100.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:38:04.851730 22895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:04.846145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:45525 (local address 127.13.100.3:45659)
0929 18:38:04.846386 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:04.846398 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:38:04.846492 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:38:04.846735 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:38:04.846737 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:38:04.846919 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:38:04.847010 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:04.847015 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:04.847831 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:04.847834 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:04.848409 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:04.848416 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:04.848883 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:04.848886 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:04.848946 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:04.849521 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:38:04.849535 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:38:04.851468 (+ 1933us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.100.62:45525: Server oryx/127.13.100.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":78,"threads_started":1}
W20260929 18:38:04.851774 22893 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:38:04.846211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.3:45659 (local address 127.13.100.62:45525)
0929 18:38:04.846330 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0929 18:38:04.846344 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 18:38:04.846449 (+ 105us) server_negotiation.cc:407] Connection header received
0929 18:38:04.846571 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:38:04.846573 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:38:04.846619 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:38:04.846699 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:38:04.847139 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:04.847713 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:04.848549 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:04.848773 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:04.848861 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:04.851684 (+ 2823us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.100.3:45659: BlockingRecv error: recv got EOF from 127.13.100.3:45659 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 29 18:38:05 dist-test-slave-j1xs krb5kdc[21883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.100.1@KRBTEST.COM for oryx/127.13.100.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:38:05 dist-test-slave-j1xs krb5kdc[21883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.100.1@KRBTEST.COM for oryx/127.13.100.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:38:05.470178 22896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:05.464587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:45525 (local address 127.13.100.1:39723)
0929 18:38:05.464868 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:05.464881 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:38:05.464969 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:38:05.465207 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:38:05.465209 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:38:05.465383 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:38:05.465475 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:05.465480 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:05.466272 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:05.466274 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:05.466810 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:05.466818 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:05.467271 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:05.467274 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:05.467364 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:05.467981 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:38:05.468002 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:38:05.470056 (+ 2054us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.100.62:45525: Server oryx/127.13.100.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":94,"threads_started":1}
W20260929 18:38:05.470506 22897 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:38:05.464705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.1:39723 (local address 127.13.100.62:45525)
0929 18:38:05.464936 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0929 18:38:05.464940 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:38:05.464952 (+ 12us) server_negotiation.cc:407] Connection header received
0929 18:38:05.465053 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:38:05.465055 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:38:05.465096 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:38:05.465175 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:38:05.465574 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:05.466182 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:05.466928 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:05.467146 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:05.467224 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:05.470386 (+ 3162us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.100.1:39723: BlockingRecv error: recv got EOF from 127.13.100.1:39723 (error 108)
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":92,"threads_started":1}
Sep 29 18:38:05 dist-test-slave-j1xs krb5kdc[21883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.100.2@KRBTEST.COM for oryx/127.13.100.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:38:05 dist-test-slave-j1xs krb5kdc[21883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.100.2@KRBTEST.COM for oryx/127.13.100.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:38:05.676823 22898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:05.667180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:45525 (local address 127.13.100.2:44417)
0929 18:38:05.667601 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:05.667616 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 18:38:05.667737 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:38:05.668063 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:38:05.668066 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:38:05.668321 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:38:05.668446 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:05.668464 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:05.669511 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:05.669514 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:05.670280 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:05.670290 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:05.670752 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:05.670755 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:05.670815 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:05.674761 (+ 3946us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:38:05.674775 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:38:05.676704 (+ 1929us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.100.62:45525: Server oryx/127.13.100.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":136,"threads_started":1}
W20260929 18:38:05.677026 22897 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:38:05.667264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.2:44417 (local address 127.13.100.62:45525)
0929 18:38:05.667444 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0929 18:38:05.667451 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 18:38:05.667721 (+ 270us) server_negotiation.cc:407] Connection header received
0929 18:38:05.667849 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:38:05.667853 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:38:05.667916 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:38:05.668004 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:38:05.668593 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:05.669394 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:05.670417 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:05.670640 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:05.670775 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:05.676848 (+ 6073us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.100.2:44417: BlockingRecv error: recv got EOF from 127.13.100.2:44417 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
Sep 29 18:38:05 dist-test-slave-j1xs krb5kdc[21883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.100.3@KRBTEST.COM for oryx/127.13.100.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:38:05 dist-test-slave-j1xs krb5kdc[21883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.100.3@KRBTEST.COM for oryx/127.13.100.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:38:05.857863 22899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:05.852570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:45525 (local address 127.13.100.3:59757)
0929 18:38:05.852834 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:05.852850 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 18:38:05.853016 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:38:05.853325 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:38:05.853328 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:38:05.853528 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:38:05.853626 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:05.853631 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:05.854351 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:05.854353 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:05.854901 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:05.854908 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:05.855317 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:05.855319 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:05.855377 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:05.855848 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:38:05.855859 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:38:05.857697 (+ 1838us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.100.62:45525: Server oryx/127.13.100.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":75,"threads_started":1}
W20260929 18:38:05.857923 22897 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:38:05.852652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.3:59757 (local address 127.13.100.62:45525)
0929 18:38:05.852774 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0929 18:38:05.852779 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:38:05.852933 (+ 154us) server_negotiation.cc:407] Connection header received
0929 18:38:05.853127 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:38:05.853131 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:38:05.853184 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:38:05.853274 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:38:05.853739 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:05.854264 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:05.855008 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:05.855224 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:05.855302 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:05.857802 (+ 2500us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.100.3:59757: BlockingRecv error: recv got EOF from 127.13.100.3:59757 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Sep 29 18:38:06 dist-test-slave-j1xs krb5kdc[21883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.100.1@KRBTEST.COM for oryx/127.13.100.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:38:06 dist-test-slave-j1xs krb5kdc[21883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.100.1@KRBTEST.COM for oryx/127.13.100.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:38:06.476202 22900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:06.470904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:45525 (local address 127.13.100.1:51021)
0929 18:38:06.471167 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:06.471185 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 18:38:06.471280 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:38:06.471518 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:38:06.471520 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:38:06.471697 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:38:06.471784 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:06.471789 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:06.472513 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:06.472515 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:06.473062 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:06.473069 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:06.473488 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:06.473492 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:06.473563 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:06.474132 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:38:06.474145 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:38:06.476085 (+ 1940us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.100.62:45525: Server oryx/127.13.100.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":83,"threads_started":1}
W20260929 18:38:06.476430 22901 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:38:06.471040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.1:51021 (local address 127.13.100.62:45525)
0929 18:38:06.471261 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0929 18:38:06.471264 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:38:06.471277 (+ 13us) server_negotiation.cc:407] Connection header received
0929 18:38:06.471358 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:38:06.471360 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:38:06.471403 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:38:06.471488 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:38:06.471887 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:06.472410 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:06.473155 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:06.473365 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:06.473437 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:06.476246 (+ 2809us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.100.1:51021: BlockingRecv error: recv got EOF from 127.13.100.1:51021 (error 108)
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":73,"threads_started":1}
Sep 29 18:38:06 dist-test-slave-j1xs krb5kdc[21883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.100.2@KRBTEST.COM for oryx/127.13.100.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:38:06 dist-test-slave-j1xs krb5kdc[21883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.100.2@KRBTEST.COM for oryx/127.13.100.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:38:06.683769 22902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:06.677536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:45525 (local address 127.13.100.2:55593)
0929 18:38:06.677797 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:06.677811 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:38:06.677934 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:38:06.678161 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:38:06.678163 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:38:06.678345 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:38:06.678440 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:06.678445 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:06.679129 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:06.679132 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:06.679683 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:06.679690 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:06.680075 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:06.680077 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:06.680134 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:06.681150 (+ 1016us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:38:06.681168 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:38:06.683574 (+ 2406us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.100.62:45525: Server oryx/127.13.100.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":63,"threads_started":1}
W20260929 18:38:06.683921 22901 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:38:06.677691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.2:55593 (local address 127.13.100.62:45525)
0929 18:38:06.677802 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0929 18:38:06.677805 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:38:06.677874 (+ 69us) server_negotiation.cc:407] Connection header received
0929 18:38:06.678004 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:38:06.678006 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:38:06.678046 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:38:06.678110 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:38:06.678535 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:06.679040 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:06.679780 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:06.679986 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:06.680057 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:06.683767 (+ 3710us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.100.2:55593: BlockingRecv error: recv got EOF from 127.13.100.2:55593 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 29 18:38:06 dist-test-slave-j1xs krb5kdc[21883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.100.3@KRBTEST.COM for oryx/127.13.100.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:38:06 dist-test-slave-j1xs krb5kdc[21883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.100.3@KRBTEST.COM for oryx/127.13.100.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:38:06.863785 22903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:06.858729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:45525 (local address 127.13.100.3:35417)
0929 18:38:06.859001 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:06.859020 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 18:38:06.859113 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:38:06.859328 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:38:06.859330 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:38:06.859524 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:38:06.859610 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:06.859615 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:06.860306 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:06.860309 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:06.860870 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:06.860877 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:06.861277 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:06.861279 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:06.861336 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:06.861878 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:38:06.861917 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:38:06.863672 (+ 1755us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.100.62:45525: Server oryx/127.13.100.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":103,"threads_started":1}
W20260929 18:38:06.863998 22901 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:38:06.858773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.3:35417 (local address 127.13.100.62:45525)
0929 18:38:06.858925 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0929 18:38:06.858930 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:38:06.859075 (+ 145us) server_negotiation.cc:407] Connection header received
0929 18:38:06.859175 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:38:06.859177 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:38:06.859221 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:38:06.859294 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:38:06.859707 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:06.860218 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:06.860975 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:06.861183 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:06.861262 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:06.863844 (+ 2582us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.100.3:35417: BlockingRecv error: recv got EOF from 127.13.100.3:35417 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 29 18:38:07 dist-test-slave-j1xs krb5kdc[21883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.100.1@KRBTEST.COM for oryx/127.13.100.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:38:07 dist-test-slave-j1xs krb5kdc[21883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.100.1@KRBTEST.COM for oryx/127.13.100.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:38:07.482587 22904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:07.477043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:45525 (local address 127.13.100.1:37255)
0929 18:38:07.477339 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:07.477358 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 18:38:07.477458 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:38:07.477783 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:38:07.477786 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:38:07.478080 (+ 294us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:38:07.478184 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:07.478191 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:07.479056 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:07.479059 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:07.479586 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:07.479593 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:07.479995 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:07.479998 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:07.480063 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:07.480592 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:38:07.480607 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:38:07.482461 (+ 1854us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.100.62:45525: Server oryx/127.13.100.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":98,"threads_started":1}
W20260929 18:38:07.482758 22905 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:38:07.477128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.1:37255 (local address 127.13.100.62:45525)
0929 18:38:07.477426 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0929 18:38:07.477429 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:38:07.477442 (+ 13us) server_negotiation.cc:407] Connection header received
0929 18:38:07.477572 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:38:07.477577 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:38:07.477648 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:38:07.477749 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:38:07.478282 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:07.478957 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:07.479687 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:07.479902 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:07.479977 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:07.482589 (+ 2612us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.100.1:37255: BlockingRecv error: recv got EOF from 127.13.100.1:37255 (error 108)
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":91,"threads_started":1}
Sep 29 18:38:07 dist-test-slave-j1xs krb5kdc[21883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.100.2@KRBTEST.COM for oryx/127.13.100.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:38:07 dist-test-slave-j1xs krb5kdc[21883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.100.2@KRBTEST.COM for oryx/127.13.100.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:38:07.690096 22906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:07.684557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:45525 (local address 127.13.100.2:46937)
0929 18:38:07.684863 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:07.684875 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:38:07.685020 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:38:07.685294 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:38:07.685297 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:38:07.685503 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:38:07.685596 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:07.685601 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:07.686449 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:07.686456 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:07.687007 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:07.687015 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:07.687479 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:07.687482 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:07.687576 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:07.688131 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:38:07.688143 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:38:07.689924 (+ 1781us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.100.62:45525: Server oryx/127.13.100.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":148,"threads_started":1}
W20260929 18:38:07.690191 22905 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:38:07.684659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.2:46937 (local address 127.13.100.62:45525)
0929 18:38:07.684808 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0929 18:38:07.684814 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:38:07.684961 (+ 147us) server_negotiation.cc:407] Connection header received
0929 18:38:07.685126 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:38:07.685130 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:38:07.685179 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:38:07.685271 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:38:07.685717 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:07.686341 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:07.687129 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:07.687368 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:07.687475 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:07.690060 (+ 2585us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.100.2:46937: BlockingRecv error: recv got EOF from 127.13.100.2:46937 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 29 18:38:07 dist-test-slave-j1xs krb5kdc[21883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.100.3@KRBTEST.COM for oryx/127.13.100.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:38:07 dist-test-slave-j1xs krb5kdc[21883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.100.3@KRBTEST.COM for oryx/127.13.100.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:38:07.869786 22907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:07.864538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:45525 (local address 127.13.100.3:51963)
0929 18:38:07.864805 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:07.864818 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:38:07.864913 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:38:07.865144 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:38:07.865146 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:38:07.865322 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:38:07.865409 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:07.865414 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:07.866131 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:07.866133 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:07.866697 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:07.866713 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:07.867113 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:07.867115 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:07.867172 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:07.867735 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:38:07.867746 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:38:07.869650 (+ 1904us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.100.62:45525: Server oryx/127.13.100.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":118,"threads_started":1}
W20260929 18:38:07.869874 22905 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:38:07.864620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.3:51963 (local address 127.13.100.62:45525)
0929 18:38:07.864798 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0929 18:38:07.864807 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 18:38:07.864888 (+ 81us) server_negotiation.cc:407] Connection header received
0929 18:38:07.864990 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:38:07.864992 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:38:07.865044 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:38:07.865110 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:38:07.865501 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:07.866034 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:07.866812 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:07.867013 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:07.867097 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:07.869785 (+ 2688us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.100.3:51963: BlockingRecv error: recv got EOF from 127.13.100.3:51963 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Sep 29 18:38:08 dist-test-slave-j1xs krb5kdc[21883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.100.1@KRBTEST.COM for oryx/127.13.100.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:38:08 dist-test-slave-j1xs krb5kdc[21883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.100.1@KRBTEST.COM for oryx/127.13.100.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:38:08.489236 22908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:08.483301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:45525 (local address 127.13.100.1:41195)
0929 18:38:08.483607 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:08.483642 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0929 18:38:08.483804 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:38:08.484056 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:38:08.484058 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:38:08.484240 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:38:08.484329 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:08.484334 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:08.485182 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:08.485185 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:08.485763 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:08.485772 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:08.486235 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:08.486238 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:08.486298 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:08.486878 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:38:08.486891 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:38:08.489123 (+ 2232us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.100.62:45525: Server oryx/127.13.100.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":85,"threads_started":1}
W20260929 18:38:08.489421 22909 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:38:08.483370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.1:41195 (local address 127.13.100.62:45525)
0929 18:38:08.483691 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0929 18:38:08.483696 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:38:08.483743 (+ 47us) server_negotiation.cc:407] Connection header received
0929 18:38:08.483890 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:38:08.483892 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:38:08.483938 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:38:08.484027 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:38:08.484438 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:08.485058 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:08.485879 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:08.486132 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:08.486215 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:08.489251 (+ 3036us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.100.1:41195: BlockingRecv error: recv got EOF from 127.13.100.1:41195 (error 108)
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":104,"threads_started":1}
Sep 29 18:38:08 dist-test-slave-j1xs krb5kdc[21883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.100.2@KRBTEST.COM for oryx/127.13.100.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:38:08 dist-test-slave-j1xs krb5kdc[21883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.100.2@KRBTEST.COM for oryx/127.13.100.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:38:08.697023 22910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:08.690882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:45525 (local address 127.13.100.2:60899)
0929 18:38:08.691249 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:08.691262 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:38:08.691391 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:38:08.691690 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:38:08.691693 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:38:08.691905 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:38:08.691992 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:08.691998 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:08.692850 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:08.692856 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:08.693470 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:08.693479 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:08.694005 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:08.694008 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:08.694098 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:08.694684 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:38:08.694696 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:38:08.696860 (+ 2164us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.100.62:45525: Server oryx/127.13.100.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":115,"threads_started":1}
W20260929 18:38:08.697257 22909 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:38:08.690941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.2:60899 (local address 127.13.100.62:45525)
0929 18:38:08.691119 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0929 18:38:08.691123 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:38:08.691340 (+ 217us) server_negotiation.cc:407] Connection header received
0929 18:38:08.691509 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:38:08.691516 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:38:08.691564 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:38:08.691669 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:38:08.692122 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:08.692744 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:08.693596 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:08.693867 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:08.694003 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:08.697039 (+ 3036us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.100.2:60899: BlockingRecv error: recv got EOF from 127.13.100.2:60899 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 29 18:38:08 dist-test-slave-j1xs krb5kdc[21883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.100.3@KRBTEST.COM for oryx/127.13.100.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:38:08 dist-test-slave-j1xs krb5kdc[21883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.100.3@KRBTEST.COM for oryx/127.13.100.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:38:08.876395 22911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:08.870615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:45525 (local address 127.13.100.3:39223)
0929 18:38:08.870945 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:08.870959 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:38:08.871059 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:38:08.871290 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:38:08.871292 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:38:08.871471 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:38:08.871595 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:08.871600 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:08.872281 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:08.872284 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:08.872901 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:08.872910 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:08.873335 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:08.873338 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:08.873395 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:08.873934 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:38:08.873947 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:38:08.876280 (+ 2333us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.100.62:45525: Server oryx/127.13.100.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":107,"threads_started":1}
W20260929 18:38:08.876559 22909 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:38:08.870659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.3:39223 (local address 127.13.100.62:45525)
0929 18:38:08.870803 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0929 18:38:08.870808 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:38:08.871034 (+ 226us) server_negotiation.cc:407] Connection header received
0929 18:38:08.871139 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:38:08.871141 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:38:08.871186 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:38:08.871250 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:38:08.871689 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:08.872188 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:08.873023 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:08.873235 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:08.873322 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:08.876404 (+ 3082us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.100.3:39223: BlockingRecv error: recv got EOF from 127.13.100.3:39223 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 29 18:38:09 dist-test-slave-j1xs krb5kdc[21883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.100.1@KRBTEST.COM for oryx/127.13.100.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:38:09 dist-test-slave-j1xs krb5kdc[21883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.100.1@KRBTEST.COM for oryx/127.13.100.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:38:09.495669 22912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:09.490000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:45525 (local address 127.13.100.1:50507)
0929 18:38:09.490297 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:09.490310 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:38:09.490396 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:38:09.490668 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:38:09.490671 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:38:09.490873 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:38:09.490967 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:09.490985 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:09.491701 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:09.491704 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:09.492269 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:09.492276 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:09.492800 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:09.492811 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:09.492892 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:09.493527 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:38:09.493543 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:38:09.495555 (+ 2012us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.100.62:45525: Server oryx/127.13.100.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":79,"threads_started":1}
W20260929 18:38:09.495841 22913 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:38:09.490114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.1:50507 (local address 127.13.100.62:45525)
0929 18:38:09.490360 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0929 18:38:09.490363 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:38:09.490376 (+ 13us) server_negotiation.cc:407] Connection header received
0929 18:38:09.490477 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:38:09.490479 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:38:09.490521 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:38:09.490606 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:38:09.491089 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:09.491607 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:09.492378 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:09.492676 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:09.492755 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:09.495675 (+ 2920us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.100.1:50507: BlockingRecv error: recv got EOF from 127.13.100.1:50507 (error 108)
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":68,"threads_started":1}
Sep 29 18:38:09 dist-test-slave-j1xs krb5kdc[21883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.100.2@KRBTEST.COM for oryx/127.13.100.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:38:09 dist-test-slave-j1xs krb5kdc[21883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.100.2@KRBTEST.COM for oryx/127.13.100.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:38:09.703826 22914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:09.697815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:45525 (local address 127.13.100.2:41079)
0929 18:38:09.698103 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:09.698126 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 18:38:09.698224 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:38:09.698495 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:38:09.698498 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:38:09.698708 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:38:09.698796 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:09.698801 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:09.699681 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:09.699686 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:09.700247 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:09.700254 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:09.700713 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:09.700716 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:09.700801 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:09.701338 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:38:09.701351 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:38:09.703630 (+ 2279us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.100.62:45525: Server oryx/127.13.100.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":97,"threads_started":1}
W20260929 18:38:09.703887 22913 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:38:09.697859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.2:41079 (local address 127.13.100.62:45525)
0929 18:38:09.698008 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0929 18:38:09.698013 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:38:09.698212 (+ 199us) server_negotiation.cc:407] Connection header received
0929 18:38:09.698331 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:38:09.698333 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:38:09.698378 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:38:09.698471 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:38:09.698912 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:09.699576 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:09.700367 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:09.700592 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:09.700704 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:09.703770 (+ 3066us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.100.2:41079: BlockingRecv error: recv got EOF from 127.13.100.2:41079 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 29 18:38:09 dist-test-slave-j1xs krb5kdc[21883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.100.3@KRBTEST.COM for oryx/127.13.100.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:38:09 dist-test-slave-j1xs krb5kdc[21883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.100.3@KRBTEST.COM for oryx/127.13.100.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:38:09.883118 22915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:09.877189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:45525 (local address 127.13.100.3:49705)
0929 18:38:09.877480 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:09.877494 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:38:09.877631 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:38:09.878043 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:38:09.878046 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:38:09.878247 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:38:09.878351 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:09.878357 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:09.879097 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:09.879100 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:09.879685 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:09.879693 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:09.880131 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:09.880134 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:09.880205 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:09.880874 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:38:09.880887 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:38:09.882993 (+ 2106us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.100.62:45525: Server oryx/127.13.100.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":107,"threads_started":1}
W20260929 18:38:09.883330 22913 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:38:09.877252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.3:49705 (local address 127.13.100.62:45525)
0929 18:38:09.877391 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0929 18:38:09.877396 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:38:09.877606 (+ 210us) server_negotiation.cc:407] Connection header received
0929 18:38:09.877783 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:38:09.877792 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:38:09.877852 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:38:09.877990 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:38:09.878467 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:09.878992 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:09.879793 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:09.879995 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:09.880069 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:09.883142 (+ 3073us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.100.3:49705: BlockingRecv error: recv got EOF from 127.13.100.3:49705 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 29 18:38:10 dist-test-slave-j1xs krb5kdc[21883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.100.1@KRBTEST.COM for oryx/127.13.100.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:38:10 dist-test-slave-j1xs krb5kdc[21883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.100.1@KRBTEST.COM for oryx/127.13.100.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:38:10.502239 22916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:10.496523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:45525 (local address 127.13.100.1:40043)
0929 18:38:10.496796 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:10.496810 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:38:10.497017 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:38:10.497299 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:38:10.497301 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:38:10.497481 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:38:10.497575 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:10.497580 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:10.498440 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:10.498444 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:10.499008 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:10.499017 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:10.499458 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:10.499461 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:10.499523 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:10.500022 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:38:10.500034 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:38:10.502046 (+ 2012us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.100.62:45525: Server oryx/127.13.100.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":100,"threads_started":1}
W20260929 18:38:10.502285 22917 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:38:10.496620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.1:40043 (local address 127.13.100.62:45525)
0929 18:38:10.496895 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0929 18:38:10.496898 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:38:10.496939 (+ 41us) server_negotiation.cc:407] Connection header received
0929 18:38:10.497134 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:38:10.497136 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:38:10.497183 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:38:10.497263 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:38:10.497710 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:10.498331 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:10.499127 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:10.499347 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:10.499430 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:10.502165 (+ 2735us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.100.1:40043: BlockingRecv error: recv got EOF from 127.13.100.1:40043 (error 108)
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":103,"threads_started":1}
Sep 29 18:38:10 dist-test-slave-j1xs krb5kdc[21883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.100.2@KRBTEST.COM for oryx/127.13.100.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:38:10 dist-test-slave-j1xs krb5kdc[21883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.100.2@KRBTEST.COM for oryx/127.13.100.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:38:10.718168 22918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:10.704584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:45525 (local address 127.13.100.2:41677)
0929 18:38:10.704818 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:10.704831 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:38:10.704920 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:38:10.705148 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:38:10.705150 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:38:10.705321 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:38:10.705408 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:10.705413 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:10.706389 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:10.706392 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:10.706964 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:10.706972 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:10.707374 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:10.707376 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:10.707441 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:10.716026 (+ 8585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:38:10.716039 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:38:10.717977 (+ 1938us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.100.62:45525: Server oryx/127.13.100.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":64,"threads_started":1}
W20260929 18:38:10.718195 22917 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:38:10.704698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.2:41677 (local address 127.13.100.62:45525)
0929 18:38:10.704871 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0929 18:38:10.704874 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:38:10.704888 (+ 14us) server_negotiation.cc:407] Connection header received
0929 18:38:10.705003 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:38:10.705006 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:38:10.705047 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:38:10.705111 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:38:10.705564 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:10.706289 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:10.707071 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:10.707279 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:10.707357 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:10.718097 (+ 10740us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.100.2:41677: BlockingRecv error: recv got EOF from 127.13.100.2:41677 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Sep 29 18:38:10 dist-test-slave-j1xs krb5kdc[21883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.100.3@KRBTEST.COM for oryx/127.13.100.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:38:10 dist-test-slave-j1xs krb5kdc[21883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.100.3@KRBTEST.COM for oryx/127.13.100.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:38:10.889385 22919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:10.883953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:45525 (local address 127.13.100.3:41355)
0929 18:38:10.884276 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:10.884289 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:38:10.884384 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:38:10.884609 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:38:10.884611 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:38:10.884784 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:38:10.884876 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:10.884880 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:10.885614 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:10.885616 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:10.886175 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:10.886182 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:10.886615 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:10.886617 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:10.886673 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:10.887164 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:38:10.887176 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:38:10.889246 (+ 2070us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.100.62:45525: Server oryx/127.13.100.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":119,"threads_started":1}
W20260929 18:38:10.889533 22917 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:38:10.884088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.3:41355 (local address 127.13.100.62:45525)
0929 18:38:10.884225 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0929 18:38:10.884228 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:38:10.884356 (+ 128us) server_negotiation.cc:407] Connection header received
0929 18:38:10.884467 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:38:10.884469 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:38:10.884516 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:38:10.884579 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:38:10.884991 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:10.885522 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:10.886295 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:10.886511 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:10.886606 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:10.889405 (+ 2799us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.100.3:41355: BlockingRecv error: recv got EOF from 127.13.100.3:41355 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 29 18:38:11 dist-test-slave-j1xs krb5kdc[21883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.100.1@KRBTEST.COM for oryx/127.13.100.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:38:11 dist-test-slave-j1xs krb5kdc[21883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.100.1@KRBTEST.COM for oryx/127.13.100.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:38:11.509429 22920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:11.503170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:45525 (local address 127.13.100.1:39203)
0929 18:38:11.503526 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:11.503541 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 18:38:11.503641 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:38:11.504118 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:38:11.504120 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:38:11.504301 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:38:11.504405 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:11.504410 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:11.505287 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:11.505289 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:11.505849 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:11.505857 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:11.506644 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:11.506648 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:11.506714 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:11.507322 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:38:11.507343 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:38:11.509257 (+ 1914us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.100.62:45525: Server oryx/127.13.100.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":107,"threads_started":1}
W20260929 18:38:11.509634 22921 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:38:11.503330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.1:39203 (local address 127.13.100.62:45525)
0929 18:38:11.503560 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0929 18:38:11.503564 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:38:11.503619 (+ 55us) server_negotiation.cc:407] Connection header received
0929 18:38:11.503846 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:38:11.503850 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:38:11.503934 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:38:11.504068 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:38:11.504552 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:11.505155 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:11.506096 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:11.506379 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:11.506627 (+ 248us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:11.509466 (+ 2839us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.100.1:39203: BlockingRecv error: recv got EOF from 127.13.100.1:39203 (error 108)
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":57,"threads_started":1}
Sep 29 18:38:11 dist-test-slave-j1xs krb5kdc[21883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.100.2@KRBTEST.COM for oryx/127.13.100.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:38:11 dist-test-slave-j1xs krb5kdc[21883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.100.2@KRBTEST.COM for oryx/127.13.100.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:38:11.724431 22922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:11.719039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:45525 (local address 127.13.100.2:55105)
0929 18:38:11.719312 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:11.719325 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:38:11.719452 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:38:11.719711 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:38:11.719714 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:38:11.719928 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:38:11.720012 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:11.720018 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:11.720886 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:11.720890 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:11.721654 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:11.721663 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:11.722188 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:11.722190 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:11.722246 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:11.722698 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:38:11.722709 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:38:11.724321 (+ 1612us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.100.62:45525: Server oryx/127.13.100.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":92,"threads_started":1}
W20260929 18:38:11.724548 22921 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:38:11.719080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.2:55105 (local address 127.13.100.62:45525)
0929 18:38:11.719224 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0929 18:38:11.719229 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:38:11.719399 (+ 170us) server_negotiation.cc:407] Connection header received
0929 18:38:11.719553 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:38:11.719556 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:38:11.719608 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:38:11.719693 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:38:11.720152 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:11.720777 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:11.721774 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:11.722088 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:11.722175 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:11.724439 (+ 2264us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.100.2:55105: BlockingRecv error: recv got EOF from 127.13.100.2:55105 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 29 18:38:11 dist-test-slave-j1xs krb5kdc[21883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.100.3@KRBTEST.COM for oryx/127.13.100.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:38:11 dist-test-slave-j1xs krb5kdc[21883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.100.3@KRBTEST.COM for oryx/127.13.100.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:38:11.895779 22923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:11.890254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:45525 (local address 127.13.100.3:50673)
0929 18:38:11.890531 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:11.890549 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 18:38:11.890638 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:38:11.890880 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:38:11.890882 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:38:11.891056 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:38:11.891160 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:11.891165 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:11.891881 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:11.891884 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:11.892449 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:11.892458 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:11.892943 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:11.892946 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:11.893008 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:11.893522 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:38:11.893533 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:38:11.895618 (+ 2085us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.100.62:45525: Server oryx/127.13.100.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":113,"threads_started":1}
W20260929 18:38:11.896057 22921 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:38:11.890412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.3:50673 (local address 127.13.100.62:45525)
0929 18:38:11.890539 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0929 18:38:11.890545 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:38:11.890601 (+ 56us) server_negotiation.cc:407] Connection header received
0929 18:38:11.890725 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:38:11.890727 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:38:11.890771 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:38:11.890845 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:38:11.891257 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:11.891764 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:11.892585 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:11.892821 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:11.892899 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:11.895761 (+ 2862us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.100.3:50673: BlockingRecv error: recv got EOF from 127.13.100.3:50673 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 29 18:38:12 dist-test-slave-j1xs krb5kdc[21883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.100.1@KRBTEST.COM for oryx/127.13.100.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:38:12 dist-test-slave-j1xs krb5kdc[21883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.100.1@KRBTEST.COM for oryx/127.13.100.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:38:12.515937 22924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:12.510443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:45525 (local address 127.13.100.1:50793)
0929 18:38:12.510701 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:12.510714 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:38:12.510815 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:38:12.511071 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:38:12.511072 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:38:12.511253 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:38:12.511348 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:12.511353 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:12.512085 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:12.512088 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:12.512672 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:12.512679 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:12.513089 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:12.513091 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:12.513157 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:12.513673 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:38:12.513684 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:38:12.515828 (+ 2144us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.100.62:45525: Server oryx/127.13.100.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":86,"threads_started":1}
W20260929 18:38:12.516100 22925 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:38:12.510498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.1:50793 (local address 127.13.100.62:45525)
0929 18:38:12.510729 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0929 18:38:12.510732 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:38:12.510784 (+ 52us) server_negotiation.cc:407] Connection header received
0929 18:38:12.510890 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:38:12.510892 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:38:12.510950 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:38:12.511026 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:38:12.511439 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:12.511973 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:12.512769 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:12.512995 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:12.513067 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:12.515966 (+ 2899us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.100.1:50793: BlockingRecv error: recv got EOF from 127.13.100.1:50793 (error 108)
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":60,"threads_started":1}
Sep 29 18:38:12 dist-test-slave-j1xs krb5kdc[21883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.100.2@KRBTEST.COM for oryx/127.13.100.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:38:12 dist-test-slave-j1xs krb5kdc[21883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.100.2@KRBTEST.COM for oryx/127.13.100.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:38:12.731055 22926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:12.725290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:45525 (local address 127.13.100.2:48309)
0929 18:38:12.725554 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:12.725566 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:38:12.725661 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:38:12.725965 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:38:12.725967 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:38:12.726156 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:38:12.726250 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:12.726256 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:12.727040 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:12.727042 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:12.727603 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:12.727611 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:12.728102 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:12.728105 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:12.728181 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:12.728708 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:38:12.728721 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:38:12.730872 (+ 2151us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.100.62:45525: Server oryx/127.13.100.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":89,"threads_started":1}
W20260929 18:38:12.731096 22925 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:38:12.725360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.2:48309 (local address 127.13.100.62:45525)
0929 18:38:12.725524 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0929 18:38:12.725527 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:38:12.725677 (+ 150us) server_negotiation.cc:407] Connection header received
0929 18:38:12.725746 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:38:12.725748 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:38:12.725792 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:38:12.725871 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:38:12.726394 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:12.726934 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:12.727724 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:12.727957 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:12.728051 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:12.731000 (+ 2949us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.100.2:48309: BlockingRecv error: recv got EOF from 127.13.100.2:48309 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 29 18:38:12 dist-test-slave-j1xs krb5kdc[21883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.100.3@KRBTEST.COM for oryx/127.13.100.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:38:12 dist-test-slave-j1xs krb5kdc[21883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.100.3@KRBTEST.COM for oryx/127.13.100.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:38:12.902889 22927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:12.896574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:45525 (local address 127.13.100.3:40663)
0929 18:38:12.896845 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:12.896858 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:38:12.896956 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:38:12.897250 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:38:12.897253 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:38:12.897453 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:38:12.897564 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:12.897570 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:12.898727 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:12.898733 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:12.899384 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:12.899398 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:12.899989 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:12.899993 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:12.900082 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:12.900720 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:38:12.900732 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:38:12.902741 (+ 2009us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.100.62:45525: Server oryx/127.13.100.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":107,"threads_started":1}
W20260929 18:38:12.903071 22925 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:38:12.896688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.3:40663 (local address 127.13.100.62:45525)
0929 18:38:12.896825 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0929 18:38:12.896828 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:38:12.896928 (+ 100us) server_negotiation.cc:407] Connection header received
0929 18:38:12.897035 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:38:12.897037 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:38:12.897083 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:38:12.897156 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:38:12.897713 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:12.898518 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:12.899573 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:12.899819 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:12.899910 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:12.902959 (+ 3049us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.100.3:40663: BlockingRecv error: recv got EOF from 127.13.100.3:40663 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 29 18:38:13 dist-test-slave-j1xs krb5kdc[21883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.100.1@KRBTEST.COM for oryx/127.13.100.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:38:13 dist-test-slave-j1xs krb5kdc[21883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.100.1@KRBTEST.COM for oryx/127.13.100.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:38:13.522295 22928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:13.516829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:45525 (local address 127.13.100.1:40007)
0929 18:38:13.517144 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:13.517157 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:38:13.517269 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:38:13.517557 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:38:13.517560 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:38:13.517762 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:38:13.517848 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:13.517868 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:13.518651 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:13.518657 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:13.519216 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:13.519223 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:13.519671 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:13.519674 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:13.519757 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:13.520302 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:38:13.520313 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:38:13.522132 (+ 1819us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.100.62:45525: Server oryx/127.13.100.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":90,"threads_started":1}
W20260929 18:38:13.522375 22929 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:38:13.516948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.1:40007 (local address 127.13.100.62:45525)
0929 18:38:13.517164 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0929 18:38:13.517168 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:38:13.517228 (+ 60us) server_negotiation.cc:407] Connection header received
0929 18:38:13.517373 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:38:13.517376 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:38:13.517431 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:38:13.517537 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:38:13.518015 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:13.518543 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:13.519335 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:13.519562 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:13.519669 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:13.522246 (+ 2577us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.100.1:40007: BlockingRecv error: recv got EOF from 127.13.100.1:40007 (error 108)
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":57,"threads_started":1}
Sep 29 18:38:13 dist-test-slave-j1xs krb5kdc[21883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.100.2@KRBTEST.COM for oryx/127.13.100.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:38:13 dist-test-slave-j1xs krb5kdc[21883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.100.2@KRBTEST.COM for oryx/127.13.100.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:38:13.737083 22930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:13.731833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:45525 (local address 127.13.100.2:39147)
0929 18:38:13.732085 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:13.732098 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:38:13.732188 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:38:13.732402 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:38:13.732404 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:38:13.732574 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:38:13.732658 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:13.732663 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:13.733343 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:13.733345 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:13.733867 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:13.733911 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:13.734365 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:13.734369 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:13.734437 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:13.734928 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:38:13.734941 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:38:13.736957 (+ 2016us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.100.62:45525: Server oryx/127.13.100.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":94,"threads_started":1}
W20260929 18:38:13.737187 22929 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:38:13.731929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.2:39147 (local address 127.13.100.62:45525)
0929 18:38:13.732071 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0929 18:38:13.732074 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:38:13.732144 (+ 70us) server_negotiation.cc:407] Connection header received
0929 18:38:13.732264 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:38:13.732266 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:38:13.732306 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:38:13.732367 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:38:13.732752 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:13.733257 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:13.734006 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:13.734237 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:13.734317 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:13.737080 (+ 2763us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.100.2:39147: BlockingRecv error: recv got EOF from 127.13.100.2:39147 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 29 18:38:13 dist-test-slave-j1xs krb5kdc[21883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.100.3@KRBTEST.COM for oryx/127.13.100.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:38:13 dist-test-slave-j1xs krb5kdc[21883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.100.3@KRBTEST.COM for oryx/127.13.100.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:38:13.909051 22931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:13.903664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:45525 (local address 127.13.100.3:34465)
0929 18:38:13.903973 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:13.903986 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:38:13.904084 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:38:13.904320 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:38:13.904322 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:38:13.904502 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:38:13.904592 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:13.904597 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:13.905310 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:13.905312 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:13.905865 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:13.905872 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:13.906361 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:13.906364 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:13.906422 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:13.906904 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:38:13.906917 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:38:13.908932 (+ 2015us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.100.62:45525: Server oryx/127.13.100.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":102,"threads_started":1}
W20260929 18:38:13.909209 22929 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:38:13.903818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.3:34465 (local address 127.13.100.62:45525)
0929 18:38:13.903958 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0929 18:38:13.903961 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:38:13.904081 (+ 120us) server_negotiation.cc:407] Connection header received
0929 18:38:13.904149 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:38:13.904151 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:38:13.904197 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:38:13.904282 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:38:13.904688 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:13.905204 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:13.906032 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:13.906255 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:13.906344 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:13.909084 (+ 2740us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.100.3:34465: BlockingRecv error: recv got EOF from 127.13.100.3:34465 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 29 18:38:14 dist-test-slave-j1xs krb5kdc[21883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.100.1@KRBTEST.COM for oryx/127.13.100.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:38:14 dist-test-slave-j1xs krb5kdc[21883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.100.1@KRBTEST.COM for oryx/127.13.100.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:38:14.528872 22932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:14.523078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:45525 (local address 127.13.100.1:39587)
0929 18:38:14.523374 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:14.523393 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 18:38:14.523518 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:38:14.523751 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:38:14.523753 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:38:14.523927 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:38:14.524016 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:14.524020 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:14.524766 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:14.524768 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:14.525401 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:14.525409 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:14.526034 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:14.526041 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:14.526130 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:14.526813 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:38:14.526826 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:38:14.528744 (+ 1918us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.100.62:45525: Server oryx/127.13.100.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":103,"threads_started":1}
W20260929 18:38:14.529013 22933 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:38:14.523183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.1:39587 (local address 127.13.100.62:45525)
0929 18:38:14.523487 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0929 18:38:14.523491 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:38:14.523523 (+ 32us) server_negotiation.cc:407] Connection header received
0929 18:38:14.523594 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:38:14.523596 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:38:14.523637 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:38:14.523715 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:38:14.524142 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:14.524672 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:14.525534 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:14.525852 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:14.526017 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:14.528860 (+ 2843us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.100.1:39587: BlockingRecv error: recv got EOF from 127.13.100.1:39587 (error 108)
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":120,"threads_started":1}
Sep 29 18:38:14 dist-test-slave-j1xs krb5kdc[21883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.100.2@KRBTEST.COM for oryx/127.13.100.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:38:14 dist-test-slave-j1xs krb5kdc[21883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.100.2@KRBTEST.COM for oryx/127.13.100.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:38:14.743300 22934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:14.738001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:45525 (local address 127.13.100.2:53359)
0929 18:38:14.738269 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:14.738282 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:38:14.738376 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:38:14.738691 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:38:14.738693 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:38:14.738868 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:38:14.738962 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:14.738967 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:14.739690 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:14.739693 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:14.740245 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:14.740252 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:14.740663 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:14.740665 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:14.740722 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:14.741202 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:38:14.741214 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:38:14.743176 (+ 1962us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.100.62:45525: Server oryx/127.13.100.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":93,"threads_started":1}
W20260929 18:38:14.743399 22933 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:38:14.738078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.2:53359 (local address 127.13.100.62:45525)
0929 18:38:14.738236 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0929 18:38:14.738240 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:38:14.738332 (+ 92us) server_negotiation.cc:407] Connection header received
0929 18:38:14.738513 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:38:14.738516 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:38:14.738577 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:38:14.738647 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:38:14.739070 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:14.739599 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:14.740348 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:14.740567 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:14.740645 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:14.743313 (+ 2668us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.100.2:53359: BlockingRecv error: recv got EOF from 127.13.100.2:53359 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 29 18:38:14 dist-test-slave-j1xs krb5kdc[21883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.100.3@KRBTEST.COM for oryx/127.13.100.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:38:14 dist-test-slave-j1xs krb5kdc[21883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.100.3@KRBTEST.COM for oryx/127.13.100.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:38:14.915838 22935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:14.910028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:45525 (local address 127.13.100.3:36573)
0929 18:38:14.910316 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:14.910328 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:38:14.910464 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:38:14.910754 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:38:14.910757 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:38:14.910969 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:38:14.911053 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:14.911059 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:14.911794 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:14.911800 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:14.912514 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:14.912523 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:14.913136 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:14.913139 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:14.913220 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:14.913703 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:38:14.913713 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:38:14.915660 (+ 1947us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.100.62:45525: Server oryx/127.13.100.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":95,"threads_started":1}
W20260929 18:38:14.915894 22933 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:38:14.910031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.3:36573 (local address 127.13.100.62:45525)
0929 18:38:14.910209 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0929 18:38:14.910213 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:38:14.910418 (+ 205us) server_negotiation.cc:407] Connection header received
0929 18:38:14.910587 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:38:14.910592 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:38:14.910648 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:38:14.910736 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:38:14.911168 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:14.911686 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:14.912640 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:14.913023 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:14.913115 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:14.915778 (+ 2663us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.100.3:36573: BlockingRecv error: recv got EOF from 127.13.100.3:36573 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 29 18:38:15 dist-test-slave-j1xs krb5kdc[21883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.100.1@KRBTEST.COM for oryx/127.13.100.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:38:15 dist-test-slave-j1xs krb5kdc[21883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.100.1@KRBTEST.COM for oryx/127.13.100.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:38:15.535895 22936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:15.529695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:45525 (local address 127.13.100.1:37583)
0929 18:38:15.530097 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:15.530116 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 18:38:15.530212 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:38:15.530487 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:38:15.530489 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:38:15.530699 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:38:15.530792 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:15.530797 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:15.531622 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:15.531625 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:15.532442 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:15.532468 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:15.533057 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:15.533061 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:15.533151 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:15.533685 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:38:15.533697 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:38:15.535750 (+ 2053us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.100.62:45525: Server oryx/127.13.100.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":111,"threads_started":1}
W20260929 18:38:15.536212 22937 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:38:15.529846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.1:37583 (local address 127.13.100.62:45525)
0929 18:38:15.530251 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0929 18:38:15.530255 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:38:15.530268 (+ 13us) server_negotiation.cc:407] Connection header received
0929 18:38:15.530302 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:38:15.530304 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:38:15.530345 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:38:15.530428 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:38:15.530959 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:15.531498 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:15.532655 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:15.532910 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:15.533000 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:15.535892 (+ 2892us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.100.1:37583: BlockingRecv error: recv got EOF from 127.13.100.1:37583 (error 108)
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":64,"threads_started":1}
Sep 29 18:38:15 dist-test-slave-j1xs krb5kdc[21883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.100.2@KRBTEST.COM for oryx/127.13.100.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:38:15 dist-test-slave-j1xs krb5kdc[21883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.100.2@KRBTEST.COM for oryx/127.13.100.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:38:15.753530 22938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:15.744196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:45525 (local address 127.13.100.2:57481)
0929 18:38:15.744467 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:15.744480 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:38:15.744576 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:38:15.744863 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:38:15.744866 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:38:15.745077 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:38:15.745180 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:15.745185 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:15.746033 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:15.746036 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:15.746965 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:15.746976 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:15.747580 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:15.747587 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:15.747679 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:15.751501 (+ 3822us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:38:15.751513 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:38:15.753407 (+ 1894us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.100.62:45525: Server oryx/127.13.100.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":91,"threads_started":1}
W20260929 18:38:15.753643 22937 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:38:15.744336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.2:57481 (local address 127.13.100.62:45525)
0929 18:38:15.744472 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0929 18:38:15.744477 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:38:15.744547 (+ 70us) server_negotiation.cc:407] Connection header received
0929 18:38:15.744661 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:38:15.744663 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:38:15.744706 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:38:15.744776 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:38:15.745315 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:15.745875 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:15.747099 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:15.747445 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:15.747555 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:15.753528 (+ 5973us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.100.2:57481: BlockingRecv error: recv got EOF from 127.13.100.2:57481 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 29 18:38:15 dist-test-slave-j1xs krb5kdc[21883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.100.3@KRBTEST.COM for oryx/127.13.100.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:38:15 dist-test-slave-j1xs krb5kdc[21883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.100.3@KRBTEST.COM for oryx/127.13.100.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:38:15.922236 22939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:15.916550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:45525 (local address 127.13.100.3:40317)
0929 18:38:15.916874 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:15.916886 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:38:15.917020 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:38:15.917292 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:38:15.917295 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:38:15.917508 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:38:15.917591 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:15.917596 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:15.918445 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:15.918448 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:15.919295 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:15.919327 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:15.919874 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:15.919877 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:15.919942 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:15.920383 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:38:15.920393 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:38:15.922117 (+ 1724us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.100.62:45525: Server oryx/127.13.100.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":83,"threads_started":1}
W20260929 18:38:15.922343 22937 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:38:15.916668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.3:40317 (local address 127.13.100.62:45525)
0929 18:38:15.916816 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0929 18:38:15.916821 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:38:15.916961 (+ 140us) server_negotiation.cc:407] Connection header received
0929 18:38:15.917127 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:38:15.917130 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:38:15.917178 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:38:15.917270 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:38:15.917706 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:15.918313 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:15.919505 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:15.919777 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:15.919857 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:15.922234 (+ 2377us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.100.3:40317: BlockingRecv error: recv got EOF from 127.13.100.3:40317 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 29 18:38:16 dist-test-slave-j1xs krb5kdc[21883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.100.1@KRBTEST.COM for oryx/127.13.100.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:38:16 dist-test-slave-j1xs krb5kdc[21883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.100.1@KRBTEST.COM for oryx/127.13.100.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:38:16.542124 22940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:16.536813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:45525 (local address 127.13.100.1:55711)
0929 18:38:16.537068 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:16.537083 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 18:38:16.537175 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:38:16.537414 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:38:16.537416 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:38:16.537596 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:38:16.537683 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:16.537689 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:16.538456 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:16.538458 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:16.539007 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:16.539015 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:16.539468 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:16.539471 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:16.539540 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:16.540059 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:38:16.540072 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:38:16.541988 (+ 1916us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.100.62:45525: Server oryx/127.13.100.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":88,"threads_started":1}
W20260929 18:38:16.542244 22941 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:38:16.536865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.1:55711 (local address 127.13.100.62:45525)
0929 18:38:16.537112 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0929 18:38:16.537115 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:38:16.537133 (+ 18us) server_negotiation.cc:407] Connection header received
0929 18:38:16.537264 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:38:16.537266 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:38:16.537306 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:38:16.537394 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:38:16.537774 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:16.538358 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:16.539114 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:16.539338 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:16.539429 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:16.542117 (+ 2688us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.100.1:55711: BlockingRecv error: recv got EOF from 127.13.100.1:55711 (error 108)
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":94,"threads_started":1}
Sep 29 18:38:16 dist-test-slave-j1xs krb5kdc[21883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.100.2@KRBTEST.COM for oryx/127.13.100.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:38:16 dist-test-slave-j1xs krb5kdc[21883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.100.2@KRBTEST.COM for oryx/127.13.100.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:38:16.759902 22942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:16.754363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:45525 (local address 127.13.100.2:55035)
0929 18:38:16.754665 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:16.754680 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 18:38:16.754810 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:38:16.755084 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:38:16.755087 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:38:16.755310 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:38:16.755395 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:16.755401 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:16.756139 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:16.756145 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:16.756696 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:16.756704 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:16.757161 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:16.757164 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:16.757250 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:16.757784 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:38:16.757795 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:38:16.759727 (+ 1932us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.100.62:45525: Server oryx/127.13.100.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":80,"threads_started":1}
W20260929 18:38:16.759963 22941 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:38:16.754467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.2:55035 (local address 127.13.100.62:45525)
0929 18:38:16.754637 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0929 18:38:16.754642 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:38:16.754762 (+ 120us) server_negotiation.cc:407] Connection header received
0929 18:38:16.754915 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:38:16.754919 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:38:16.754971 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:38:16.755056 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:38:16.755510 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:16.756030 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:16.756821 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:16.757046 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:16.757159 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:16.759853 (+ 2694us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.100.2:55035: BlockingRecv error: recv got EOF from 127.13.100.2:55035 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 18:38:16.821051 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 22466
I20260929 18:38:16.826543 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 22616
I20260929 18:38:16.832190 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 22750
I20260929 18:38:16.837400 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 22396
2026-09-29T18:38:16Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (18002 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-29T18:38:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:38:16Z Disabled control of system clock
I20260929 18:38:16.867468 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707036041078-13712-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707036041078-13712-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707036041078-13712-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.100.62:43449
--webserver_interface=127.13.100.62
--webserver_port=0
--builtin_ntp_servers=127.13.100.20:43619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.100.62:43449
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260929 18:38:16.973428 22946 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:38:16.973655 22946 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:38:16.973688 22946 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:38:16.977667 22946 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:38:16.977720 22946 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:38:16.977736 22946 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:38:16.977753 22946 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:38:16.977805 22946 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:38:16.982460 22946 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.100.20:43619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707036041078-13712-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707036041078-13712-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.100.62:43449
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.100.62:43449
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.100.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22946
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707036041078-13712-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:38:16.983469 22946 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:38:16.984686 22946 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:38:16.992764 22952 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:38:16.992816 22951 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:38:16.992851 22954 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:38:16.992916 22946 server_base.cc:1034] running on GCE node
I20260929 18:38:16.993543 22946 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:38:16.994596 22946 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:38:16.995774 22946 hybrid_clock.cc:648] HybridClock initialized: now 1790707096995751 us; error 45 us; skew 500 ppm
I20260929 18:38:16.997579 22946 webserver.cc:460] Webserver started at http://127.13.100.62:35509/ using document root <none> and password file <none>
I20260929 18:38:16.998188 22946 fs_manager.cc:359] Metadata directory not provided
I20260929 18:38:16.998275 22946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:38:16.998495 22946 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:38:17.000391 22946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707036041078-13712-0/minicluster-data/master-0/data/instance:
uuid: "76f5339c07f94be0bd783df29a95e152"
format_stamp: "Formatted at 2026-09-29 18:38:16 on dist-test-slave-j1xs"
server_key: "9e1d3a23c7b242bb18934cf8250afe26"
server_key_iv: "c2ac6c8e29d393bffe7f6bb5f4a5cbb5"
server_key_version: "encryptionkey@0"
I20260929 18:38:17.000929 22946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707036041078-13712-0/minicluster-data/master-0/wal/instance:
uuid: "76f5339c07f94be0bd783df29a95e152"
format_stamp: "Formatted at 2026-09-29 18:38:16 on dist-test-slave-j1xs"
server_key: "9e1d3a23c7b242bb18934cf8250afe26"
server_key_iv: "c2ac6c8e29d393bffe7f6bb5f4a5cbb5"
server_key_version: "encryptionkey@0"
I20260929 18:38:17.004429 22946 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260929 18:38:17.006690 22960 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:38:17.007833 22946 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:38:17.007985 22946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707036041078-13712-0/minicluster-data/master-0/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707036041078-13712-0/minicluster-data/master-0/wal
uuid: "76f5339c07f94be0bd783df29a95e152"
format_stamp: "Formatted at 2026-09-29 18:38:16 on dist-test-slave-j1xs"
server_key: "9e1d3a23c7b242bb18934cf8250afe26"
server_key_iv: "c2ac6c8e29d393bffe7f6bb5f4a5cbb5"
server_key_version: "encryptionkey@0"
I20260929 18:38:17.008087 22946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707036041078-13712-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707036041078-13712-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707036041078-13712-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:38:17.017848 22946 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:38:17.018474 22946 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:38:17.018648 22946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:38:17.025025 22946 rpc_server.cc:307] RPC server started. Bound to: 127.13.100.62:43449
I20260929 18:38:17.025033 23012 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.100.62:43449 every 8 connection(s)
I20260929 18:38:17.025869 22946 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
I20260929 18:38:17.028771 23013 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:38:17.032809 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 22946
I20260929 18:38:17.032904 13712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707036041078-13712-0/minicluster-data/master-0/wal/instance
I20260929 18:38:17.033200 13712 external_mini_cluster.cc:1428] Setting key b4371009ed98689132b966d20f20d40c
I20260929 18:38:17.034878 23013 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76f5339c07f94be0bd783df29a95e152: Bootstrap starting.
I20260929 18:38:17.037005 23013 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 76f5339c07f94be0bd783df29a95e152: Neither blocks nor log segments found. Creating new log.
I20260929 18:38:17.037681 23013 log.cc:824] T 00000000000000000000000000000000 P 76f5339c07f94be0bd783df29a95e152: Log is configured to *not* fsync() on all Append() calls
I20260929 18:38:17.039646 23013 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76f5339c07f94be0bd783df29a95e152: No bootstrap required, opened a new log
I20260929 18:38:17.041658 23016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:17.034491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33674 (local address 127.13.100.62:43449)
0929 18:38:17.035073 (+ 582us) server_negotiation.cc:206] Beginning negotiation
0929 18:38:17.035087 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 18:38:17.035121 (+ 34us) server_negotiation.cc:407] Connection header received
0929 18:38:17.035711 (+ 590us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:38:17.035726 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:38:17.036112 (+ 386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:38:17.036374 (+ 262us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:38:17.036678 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:17.037644 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:17.038730 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:17.039150 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:17.039266 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:17.040109 (+ 843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:38:17.040114 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:38:17.040125 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:38:17.040486 (+ 361us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:38:17.040534 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:38:17.040552 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:38:17.040625 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 18:38:17.040780 (+ 155us) server_negotiation.cc:299] Negotiation successful
0929 18:38:17.041040 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240,"thread_start_us":98,"threads_started":1}
I20260929 18:38:17.043548 23013 raft_consensus.cc:348] T 00000000000000000000000000000000 P 76f5339c07f94be0bd783df29a95e152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76f5339c07f94be0bd783df29a95e152" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 43449 } }
I20260929 18:38:17.043810 23013 raft_consensus.cc:374] T 00000000000000000000000000000000 P 76f5339c07f94be0bd783df29a95e152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:38:17.043886 23013 raft_consensus.cc:729] T 00000000000000000000000000000000 P 76f5339c07f94be0bd783df29a95e152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76f5339c07f94be0bd783df29a95e152, State: Initialized, Role: FOLLOWER
I20260929 18:38:17.044312 23013 consensus_queue.cc:260] T 00000000000000000000000000000000 P 76f5339c07f94be0bd783df29a95e152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76f5339c07f94be0bd783df29a95e152" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 43449 } }
I20260929 18:38:17.044456 23013 raft_consensus.cc:388] T 00000000000000000000000000000000 P 76f5339c07f94be0bd783df29a95e152 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:38:17.044540 23013 raft_consensus.cc:482] T 00000000000000000000000000000000 P 76f5339c07f94be0bd783df29a95e152 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:38:17.044632 23013 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 76f5339c07f94be0bd783df29a95e152 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:38:17.045452 23013 raft_consensus.cc:504] T 00000000000000000000000000000000 P 76f5339c07f94be0bd783df29a95e152 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76f5339c07f94be0bd783df29a95e152" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 43449 } }
I20260929 18:38:17.045768 23013 leader_election.cc:302] T 00000000000000000000000000000000 P 76f5339c07f94be0bd783df29a95e152 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 76f5339c07f94be0bd783df29a95e152; no voters:
I20260929 18:38:17.046082 23013 leader_election.cc:288] T 00000000000000000000000000000000 P 76f5339c07f94be0bd783df29a95e152 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:38:17.046177 23018 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 76f5339c07f94be0bd783df29a95e152 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:38:17.046458 23018 raft_consensus.cc:686] T 00000000000000000000000000000000 P 76f5339c07f94be0bd783df29a95e152 [term 1 LEADER]: Becoming Leader. State: Replica: 76f5339c07f94be0bd783df29a95e152, State: Running, Role: LEADER
I20260929 18:38:17.046787 23018 consensus_queue.cc:237] T 00000000000000000000000000000000 P 76f5339c07f94be0bd783df29a95e152 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76f5339c07f94be0bd783df29a95e152" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 43449 } }
I20260929 18:38:17.047245 23013 sys_catalog.cc:564] T 00000000000000000000000000000000 P 76f5339c07f94be0bd783df29a95e152 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:38:17.048338 23019 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76f5339c07f94be0bd783df29a95e152 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "76f5339c07f94be0bd783df29a95e152" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76f5339c07f94be0bd783df29a95e152" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 43449 } } }
I20260929 18:38:17.048478 23019 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76f5339c07f94be0bd783df29a95e152 [sys.catalog]: This master's current role is: LEADER
I20260929 18:38:17.048761 23020 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76f5339c07f94be0bd783df29a95e152 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 76f5339c07f94be0bd783df29a95e152. Latest consensus state: current_term: 1 leader_uuid: "76f5339c07f94be0bd783df29a95e152" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76f5339c07f94be0bd783df29a95e152" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 43449 } } }
I20260929 18:38:17.048861 23020 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76f5339c07f94be0bd783df29a95e152 [sys.catalog]: This master's current role is: LEADER
I20260929 18:38:17.048851 23027 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:38:17.051553 23027 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:38:17.056723 23027 catalog_manager.cc:1347] Generated new cluster ID: 85fe46717d924a52a0850c8d06f29ae2
I20260929 18:38:17.056785 23027 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:38:17.078696 23027 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:38:17.079926 23027 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:38:17.087836 23027 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 76f5339c07f94be0bd783df29a95e152: Generated new TSK 0
I20260929 18:38:17.088639 23027 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:38:17.100288 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707036041078-13712-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707036041078-13712-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707036041078-13712-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707036041078-13712-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.100.1:0
--local_ip_for_outbound_sockets=127.13.100.1
--webserver_interface=127.13.100.1
--webserver_port=0
--tserver_master_addrs=127.13.100.62:43449
--builtin_ntp_servers=127.13.100.20:43619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260929 18:38:17.213837 23037 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:38:17.214148 23037 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:38:17.214215 23037 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:38:17.218178 23037 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:38:17.218255 23037 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:38:17.218340 23037 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.100.1
I20260929 18:38:17.223256 23037 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.100.20:43619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707036041078-13712-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707036041078-13712-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.100.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707036041078-13712-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.100.1
--webserver_port=0
--tserver_master_addrs=127.13.100.62:43449
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23037
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.100.1
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707036041078-13712-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:38:17.224339 23037 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:38:17.225713 23037 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:38:17.234900 23042 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:38:17.234927 23043 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:38:17.234925 23045 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:38:17.235302 23037 server_base.cc:1034] running on GCE node
I20260929 18:38:17.235635 23037 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:38:17.236207 23037 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:38:17.237394 23037 hybrid_clock.cc:648] HybridClock initialized: now 1790707097237365 us; error 53 us; skew 500 ppm
I20260929 18:38:17.239310 23037 webserver.cc:460] Webserver started at http://127.13.100.1:46691/ using document root <none> and password file <none>
I20260929 18:38:17.239868 23037 fs_manager.cc:359] Metadata directory not provided
I20260929 18:38:17.239969 23037 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:38:17.240192 23037 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:38:17.242311 23037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707036041078-13712-0/minicluster-data/ts-0/data/instance:
uuid: "f541e50102b94003ae9d7fef2c29bc98"
format_stamp: "Formatted at 2026-09-29 18:38:17 on dist-test-slave-j1xs"
server_key: "4f32815fe90b82fce19a8dd0fd50ee08"
server_key_iv: "7054907df711462cb8c2c7a76f42dccf"
server_key_version: "encryptionkey@0"
I20260929 18:38:17.242854 23037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707036041078-13712-0/minicluster-data/ts-0/wal/instance:
uuid: "f541e50102b94003ae9d7fef2c29bc98"
format_stamp: "Formatted at 2026-09-29 18:38:17 on dist-test-slave-j1xs"
server_key: "4f32815fe90b82fce19a8dd0fd50ee08"
server_key_iv: "7054907df711462cb8c2c7a76f42dccf"
server_key_version: "encryptionkey@0"
I20260929 18:38:17.246456 23037 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260929 18:38:17.249019 23051 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:38:17.250238 23037 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:38:17.250367 23037 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707036041078-13712-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707036041078-13712-0/minicluster-data/ts-0/wal
uuid: "f541e50102b94003ae9d7fef2c29bc98"
format_stamp: "Formatted at 2026-09-29 18:38:17 on dist-test-slave-j1xs"
server_key: "4f32815fe90b82fce19a8dd0fd50ee08"
server_key_iv: "7054907df711462cb8c2c7a76f42dccf"
server_key_version: "encryptionkey@0"
I20260929 18:38:17.250484 23037 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707036041078-13712-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707036041078-13712-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707036041078-13712-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:38:17.264750 23037 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:38:17.265441 23037 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:38:17.265646 23037 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:38:17.266274 23037 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:38:17.267180 23037 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:38:17.267253 23037 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:38:17.267323 23037 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:38:17.267373 23037 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:38:17.276068 23037 rpc_server.cc:307] RPC server started. Bound to: 127.13.100.1:39917
I20260929 18:38:17.276091 23164 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.100.1:39917 every 8 connection(s)
I20260929 18:38:17.276919 23037 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707036041078-13712-0/minicluster-data/ts-0/data/info.pb
I20260929 18:38:17.285636 23016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:17.278607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.1:55183 (local address 127.13.100.62:43449)
0929 18:38:17.278740 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0929 18:38:17.278744 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:38:17.279746 (+ 1002us) server_negotiation.cc:407] Connection header received
0929 18:38:17.281095 (+ 1349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:38:17.281098 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:38:17.281153 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:38:17.281254 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:38:17.282128 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:17.282652 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:17.283778 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:17.284006 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:17.284135 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:17.285158 (+ 1023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:38:17.285162 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:38:17.285164 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:38:17.285176 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:38:17.285203 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:38:17.285211 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:38:17.285286 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 18:38:17.285443 (+ 157us) server_negotiation.cc:299] Negotiation successful
0929 18:38:17.285498 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 18:38:17.286521 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 23037
I20260929 18:38:17.286571 23167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:17.279001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:43449 (local address 127.13.100.1:55183)
0929 18:38:17.279608 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:17.279645 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 18:38:17.280882 (+ 1237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:38:17.281408 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:38:17.281415 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:38:17.281490 (+ 75us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:38:17.281993 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:17.282006 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:17.282771 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:17.282776 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:17.283655 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:17.283664 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:17.284168 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:17.284172 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:17.284327 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:17.284918 (+ 591us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:38:17.284919 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:38:17.284955 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:38:17.285011 (+ 56us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:38:17.285299 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:38:17.285308 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:38:17.285314 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 18:38:17.285441 (+ 127us) client_negotiation.cc:240] Negotiation successful
0929 18:38:17.285705 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":108,"threads_started":1}
I20260929 18:38:17.286643 13712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707036041078-13712-0/minicluster-data/ts-0/wal/instance
I20260929 18:38:17.286945 13712 external_mini_cluster.cc:1428] Setting key 6518ab75c321a8d6cbb0a7fad77ac422
I20260929 18:38:17.287799 23165 heartbeater.cc:347] Connected to a master server at 127.13.100.62:43449
I20260929 18:38:17.288010 23165 heartbeater.cc:464] Registering TS with master...
I20260929 18:38:17.288594 23165 heartbeater.cc:511] Master 127.13.100.62:43449 requested a full tablet report, sending...
I20260929 18:38:17.288892 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707036041078-13712-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707036041078-13712-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707036041078-13712-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707036041078-13712-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.100.2:0
--local_ip_for_outbound_sockets=127.13.100.2
--webserver_interface=127.13.100.2
--webserver_port=0
--tserver_master_addrs=127.13.100.62:43449
--builtin_ntp_servers=127.13.100.20:43619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260929 18:38:17.290151 22977 ts_manager.cc:194] Registered new tserver with Master: f541e50102b94003ae9d7fef2c29bc98 (127.13.100.1:39917)
I20260929 18:38:17.291795 22977 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.100.1:55183
W20260929 18:38:17.398231 23168 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:38:17.398473 23168 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:38:17.398510 23168 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:38:17.402454 23168 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:38:17.402503 23168 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:38:17.402555 23168 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.100.2
I20260929 18:38:17.407238 23168 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.100.20:43619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707036041078-13712-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707036041078-13712-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.100.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707036041078-13712-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.100.2
--webserver_port=0
--tserver_master_addrs=127.13.100.62:43449
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23168
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.100.2
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707036041078-13712-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:38:17.408214 23168 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:38:17.409472 23168 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:38:17.418808 23176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:38:17.418798 23173 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:38:17.418798 23174 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:38:17.419270 23168 server_base.cc:1034] running on GCE node
I20260929 18:38:17.419597 23168 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:38:17.420125 23168 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:38:17.421280 23168 hybrid_clock.cc:648] HybridClock initialized: now 1790707097421265 us; error 28 us; skew 500 ppm
I20260929 18:38:17.423039 23168 webserver.cc:460] Webserver started at http://127.13.100.2:34387/ using document root <none> and password file <none>
I20260929 18:38:17.423604 23168 fs_manager.cc:359] Metadata directory not provided
I20260929 18:38:17.423663 23168 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:38:17.423862 23168 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:38:17.425830 23168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707036041078-13712-0/minicluster-data/ts-1/data/instance:
uuid: "65b57f20487b4ab9b880ee82138b510e"
format_stamp: "Formatted at 2026-09-29 18:38:17 on dist-test-slave-j1xs"
server_key: "0e8cd3f7aadd170290efdba645133723"
server_key_iv: "2bde635c8fe40d1b7cd8c91339b01c5d"
server_key_version: "encryptionkey@0"
I20260929 18:38:17.426430 23168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707036041078-13712-0/minicluster-data/ts-1/wal/instance:
uuid: "65b57f20487b4ab9b880ee82138b510e"
format_stamp: "Formatted at 2026-09-29 18:38:17 on dist-test-slave-j1xs"
server_key: "0e8cd3f7aadd170290efdba645133723"
server_key_iv: "2bde635c8fe40d1b7cd8c91339b01c5d"
server_key_version: "encryptionkey@0"
I20260929 18:38:17.429862 23168 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:38:17.432261 23182 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:38:17.433377 23168 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:38:17.433485 23168 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707036041078-13712-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707036041078-13712-0/minicluster-data/ts-1/wal
uuid: "65b57f20487b4ab9b880ee82138b510e"
format_stamp: "Formatted at 2026-09-29 18:38:17 on dist-test-slave-j1xs"
server_key: "0e8cd3f7aadd170290efdba645133723"
server_key_iv: "2bde635c8fe40d1b7cd8c91339b01c5d"
server_key_version: "encryptionkey@0"
I20260929 18:38:17.433586 23168 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707036041078-13712-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707036041078-13712-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707036041078-13712-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:38:17.450837 23168 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:38:17.451445 23168 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:38:17.451642 23168 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:38:17.452189 23168 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:38:17.453044 23168 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:38:17.453128 23168 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:38:17.453209 23168 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:38:17.453239 23168 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:38:17.462134 23168 rpc_server.cc:307] RPC server started. Bound to: 127.13.100.2:43069
I20260929 18:38:17.462224 23295 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.100.2:43069 every 8 connection(s)
I20260929 18:38:17.463032 23168 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707036041078-13712-0/minicluster-data/ts-1/data/info.pb
I20260929 18:38:17.465492 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 23168
I20260929 18:38:17.465636 13712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707036041078-13712-0/minicluster-data/ts-1/wal/instance
I20260929 18:38:17.466012 13712 external_mini_cluster.cc:1428] Setting key 24a6f9dd80f73d28bac5f18c6f391d09
I20260929 18:38:17.467993 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707036041078-13712-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707036041078-13712-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707036041078-13712-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707036041078-13712-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.100.3:0
--local_ip_for_outbound_sockets=127.13.100.3
--webserver_interface=127.13.100.3
--webserver_port=0
--tserver_master_addrs=127.13.100.62:43449
--builtin_ntp_servers=127.13.100.20:43619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260929 18:38:17.473166 23016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:17.464830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.2:56065 (local address 127.13.100.62:43449)
0929 18:38:17.465018 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0929 18:38:17.465021 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:38:17.466070 (+ 1049us) server_negotiation.cc:407] Connection header received
0929 18:38:17.467593 (+ 1523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:38:17.467597 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:38:17.467644 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:38:17.467733 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:38:17.468631 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:17.469454 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:17.471209 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:17.471409 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:17.471492 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:17.472728 (+ 1236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:38:17.472731 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:38:17.472733 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:38:17.472745 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:38:17.472763 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:38:17.472770 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:38:17.472827 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0929 18:38:17.473012 (+ 185us) server_negotiation.cc:299] Negotiation successful
0929 18:38:17.473059 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260929 18:38:17.474473 23298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:17.465178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:43449 (local address 127.13.100.2:56065)
0929 18:38:17.465903 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:17.465955 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 18:38:17.467401 (+ 1446us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:38:17.467878 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:38:17.467884 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:38:17.467961 (+ 77us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:38:17.468494 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:17.468506 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:17.469578 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:17.469583 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:17.471089 (+ 1506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:17.471103 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:17.471556 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:17.471563 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:17.471783 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:17.472425 (+ 642us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:38:17.472430 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:38:17.472487 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:38:17.472573 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:38:17.472853 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:38:17.472863 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:38:17.472873 (+ 10us) client_negotiation.cc:769] Sending connection context
0929 18:38:17.473012 (+ 139us) client_negotiation.cc:240] Negotiation successful
0929 18:38:17.473337 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":134,"threads_started":1}
I20260929 18:38:17.475766 23296 heartbeater.cc:347] Connected to a master server at 127.13.100.62:43449
I20260929 18:38:17.476078 23296 heartbeater.cc:464] Registering TS with master...
I20260929 18:38:17.476627 23296 heartbeater.cc:511] Master 127.13.100.62:43449 requested a full tablet report, sending...
I20260929 18:38:17.477736 22977 ts_manager.cc:194] Registered new tserver with Master: 65b57f20487b4ab9b880ee82138b510e (127.13.100.2:43069)
I20260929 18:38:17.478623 22977 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.100.2:56065
W20260929 18:38:17.577402 23299 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:38:17.577636 23299 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:38:17.577672 23299 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:38:17.581552 23299 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:38:17.581612 23299 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:38:17.581664 23299 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.100.3
I20260929 18:38:17.586309 23299 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.100.20:43619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707036041078-13712-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707036041078-13712-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.100.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707036041078-13712-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.100.3
--webserver_port=0
--tserver_master_addrs=127.13.100.62:43449
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23299
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.100.3
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707036041078-13712-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:38:17.587347 23299 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:38:17.588963 23299 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 18:38:17.598868 23299 server_base.cc:1034] running on GCE node
W20260929 18:38:17.598802 23305 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:38:17.598810 23307 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:38:17.598810 23304 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:38:17.599536 23299 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:38:17.600086 23299 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:38:17.601243 23299 hybrid_clock.cc:648] HybridClock initialized: now 1790707097601213 us; error 43 us; skew 500 ppm
I20260929 18:38:17.603065 23299 webserver.cc:460] Webserver started at http://127.13.100.3:45687/ using document root <none> and password file <none>
I20260929 18:38:17.603607 23299 fs_manager.cc:359] Metadata directory not provided
I20260929 18:38:17.603672 23299 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:38:17.603888 23299 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:38:17.605947 23299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707036041078-13712-0/minicluster-data/ts-2/data/instance:
uuid: "4b050b086bfb4c309e4ba7f112b7a35e"
format_stamp: "Formatted at 2026-09-29 18:38:17 on dist-test-slave-j1xs"
server_key: "532cfbed9ca9d5201d5c90d86ee7fa19"
server_key_iv: "e3de0737dadb12a42a63832c817cd5c4"
server_key_version: "encryptionkey@0"
I20260929 18:38:17.606528 23299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707036041078-13712-0/minicluster-data/ts-2/wal/instance:
uuid: "4b050b086bfb4c309e4ba7f112b7a35e"
format_stamp: "Formatted at 2026-09-29 18:38:17 on dist-test-slave-j1xs"
server_key: "532cfbed9ca9d5201d5c90d86ee7fa19"
server_key_iv: "e3de0737dadb12a42a63832c817cd5c4"
server_key_version: "encryptionkey@0"
I20260929 18:38:17.610131 23299 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260929 18:38:17.612632 23313 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:38:17.613835 23299 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:38:17.614025 23299 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707036041078-13712-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707036041078-13712-0/minicluster-data/ts-2/wal
uuid: "4b050b086bfb4c309e4ba7f112b7a35e"
format_stamp: "Formatted at 2026-09-29 18:38:17 on dist-test-slave-j1xs"
server_key: "532cfbed9ca9d5201d5c90d86ee7fa19"
server_key_iv: "e3de0737dadb12a42a63832c817cd5c4"
server_key_version: "encryptionkey@0"
I20260929 18:38:17.614151 23299 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707036041078-13712-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707036041078-13712-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707036041078-13712-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:38:17.645619 23299 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:38:17.646414 23299 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:38:17.646636 23299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:38:17.647229 23299 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:38:17.648319 23299 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:38:17.648397 23299 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:38:17.648486 23299 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:38:17.648542 23299 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:38:17.657648 23299 rpc_server.cc:307] RPC server started. Bound to: 127.13.100.3:41369
I20260929 18:38:17.657693 23426 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.100.3:41369 every 8 connection(s)
I20260929 18:38:17.658528 23299 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707036041078-13712-0/minicluster-data/ts-2/data/info.pb
I20260929 18:38:17.664956 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 23299
I20260929 18:38:17.665103 13712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707036041078-13712-0/minicluster-data/ts-2/wal/instance
I20260929 18:38:17.665395 13712 external_mini_cluster.cc:1428] Setting key 7906d1c7b683ff0a3776baf244cdd033
I20260929 18:38:17.667721 23016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:17.660193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.3:47809 (local address 127.13.100.62:43449)
0929 18:38:17.660342 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0929 18:38:17.660346 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:38:17.661312 (+ 966us) server_negotiation.cc:407] Connection header received
0929 18:38:17.662668 (+ 1356us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:38:17.662671 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:38:17.662717 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:38:17.662805 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:38:17.663645 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:17.664196 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:17.665420 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:17.665733 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:17.665828 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:17.667220 (+ 1392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:38:17.667223 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:38:17.667225 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:38:17.667236 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:38:17.667255 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:38:17.667262 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:38:17.667324 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0929 18:38:17.667496 (+ 172us) server_negotiation.cc:299] Negotiation successful
0929 18:38:17.667575 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 18:38:17.668556 23429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:17.660578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:43449 (local address 127.13.100.3:47809)
0929 18:38:17.661173 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:17.661211 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 18:38:17.662463 (+ 1252us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:38:17.662967 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:38:17.662972 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:38:17.663048 (+ 76us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:38:17.663501 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:17.663513 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:17.664325 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:17.664328 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:17.665299 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:17.665309 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:17.665943 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:17.665951 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:17.666183 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:17.666918 (+ 735us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:38:17.666920 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:38:17.666964 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:38:17.667061 (+ 97us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:38:17.667349 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:38:17.667357 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:38:17.667363 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 18:38:17.667490 (+ 127us) client_negotiation.cc:240] Negotiation successful
0929 18:38:17.667771 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":105,"threads_started":1}
I20260929 18:38:17.669719 23427 heartbeater.cc:347] Connected to a master server at 127.13.100.62:43449
I20260929 18:38:17.669983 23427 heartbeater.cc:464] Registering TS with master...
I20260929 18:38:17.670485 23427 heartbeater.cc:511] Master 127.13.100.62:43449 requested a full tablet report, sending...
I20260929 18:38:17.671455 22977 ts_manager.cc:194] Registered new tserver with Master: 4b050b086bfb4c309e4ba7f112b7a35e (127.13.100.3:41369)
I20260929 18:38:17.672014 22977 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.100.3:47809
I20260929 18:38:17.679383 13712 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260929 18:38:17.682168 23016 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:38:17.681207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33684 (local address 127.13.100.62:43449)
0929 18:38:17.681340 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0929 18:38:17.681345 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:38:17.681503 (+ 158us) server_negotiation.cc:407] Connection header received
0929 18:38:17.681651 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:38:17.681654 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:38:17.681698 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:38:17.681789 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:38:17.682054 (+ 265us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33684: BlockingRecv error: recv got EOF from 127.0.0.1:33684 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 18:38:17.682330 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 23037
I20260929 18:38:17.687508 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 23168
I20260929 18:38:17.692373 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 23299
I20260929 18:38:17.697510 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 22946
2026-09-29T18:38:17Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (858 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-29T18:38:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:38:17Z Disabled control of system clock
I20260929 18:38:17.725850 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707036041078-13712-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707036041078-13712-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707036041078-13712-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.100.62:43117
--webserver_interface=127.13.100.62
--webserver_port=0
--builtin_ntp_servers=127.13.100.20:39109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.100.62:43117
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260929 18:38:17.834986 23438 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:38:17.835260 23438 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:38:17.835299 23438 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:38:17.839330 23438 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:38:17.839421 23438 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:38:17.839439 23438 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:38:17.839457 23438 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:38:17.839473 23438 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:38:17.844493 23438 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.100.20:39109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707036041078-13712-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707036041078-13712-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.100.62:43117
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.100.62:43117
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.100.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23438
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707036041078-13712-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:38:17.845557 23438 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:38:17.846925 23438 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:38:17.855053 23444 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:38:17.855049 23443 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:38:17.855283 23438 server_base.cc:1034] running on GCE node
W20260929 18:38:17.855036 23446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:38:17.855928 23438 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:38:17.857033 23438 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:38:17.858173 23438 hybrid_clock.cc:648] HybridClock initialized: now 1790707097858145 us; error 43 us; skew 500 ppm
I20260929 18:38:17.859908 23438 webserver.cc:460] Webserver started at http://127.13.100.62:43003/ using document root <none> and password file <none>
I20260929 18:38:17.860464 23438 fs_manager.cc:359] Metadata directory not provided
I20260929 18:38:17.860555 23438 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:38:17.860787 23438 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:38:17.862821 23438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707036041078-13712-0/minicluster-data/master-0/data/instance:
uuid: "adc2adc306ab41da812fad0232231889"
format_stamp: "Formatted at 2026-09-29 18:38:17 on dist-test-slave-j1xs"
server_key: "82ea78f4b5553be48754e7cba3c99497"
server_key_iv: "62c8048005b8808338ee645f4fc79c6e"
server_key_version: "encryptionkey@0"
I20260929 18:38:17.863380 23438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707036041078-13712-0/minicluster-data/master-0/wal/instance:
uuid: "adc2adc306ab41da812fad0232231889"
format_stamp: "Formatted at 2026-09-29 18:38:17 on dist-test-slave-j1xs"
server_key: "82ea78f4b5553be48754e7cba3c99497"
server_key_iv: "62c8048005b8808338ee645f4fc79c6e"
server_key_version: "encryptionkey@0"
I20260929 18:38:17.866967 23438 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260929 18:38:17.869279 23452 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:38:17.870726 23438 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260929 18:38:17.870873 23438 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707036041078-13712-0/minicluster-data/master-0/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707036041078-13712-0/minicluster-data/master-0/wal
uuid: "adc2adc306ab41da812fad0232231889"
format_stamp: "Formatted at 2026-09-29 18:38:17 on dist-test-slave-j1xs"
server_key: "82ea78f4b5553be48754e7cba3c99497"
server_key_iv: "62c8048005b8808338ee645f4fc79c6e"
server_key_version: "encryptionkey@0"
I20260929 18:38:17.870992 23438 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707036041078-13712-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707036041078-13712-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707036041078-13712-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:38:17.875340 23438 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:38:17.875916 23438 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:38:17.876102 23438 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:38:17.882709 23438 rpc_server.cc:307] RPC server started. Bound to: 127.13.100.62:43117
I20260929 18:38:17.882720 23504 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.100.62:43117 every 8 connection(s)
I20260929 18:38:17.883628 23438 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
I20260929 18:38:17.886739 23505 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:38:17.891371 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 23438
I20260929 18:38:17.891494 13712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707036041078-13712-0/minicluster-data/master-0/wal/instance
I20260929 18:38:17.891794 13712 external_mini_cluster.cc:1428] Setting key a8c052de9f7f11cead7ecde189e3bebd
I20260929 18:38:17.892537 23505 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P adc2adc306ab41da812fad0232231889: Bootstrap starting.
I20260929 18:38:17.895049 23505 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P adc2adc306ab41da812fad0232231889: Neither blocks nor log segments found. Creating new log.
I20260929 18:38:17.895859 23505 log.cc:824] T 00000000000000000000000000000000 P adc2adc306ab41da812fad0232231889: Log is configured to *not* fsync() on all Append() calls
I20260929 18:38:17.897423 23508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:17.893233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49756 (local address 127.13.100.62:43117)
0929 18:38:17.894046 (+ 813us) server_negotiation.cc:206] Beginning negotiation
0929 18:38:17.894063 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 18:38:17.894126 (+ 63us) server_negotiation.cc:407] Connection header received
0929 18:38:17.894742 (+ 616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:38:17.894757 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:38:17.895123 (+ 366us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:38:17.895462 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:38:17.896250 (+ 788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:38:17.896254 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:38:17.896265 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:38:17.896275 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:38:17.896327 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:38:17.896341 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:38:17.896455 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0929 18:38:17.896551 (+ 96us) server_negotiation.cc:299] Negotiation successful
0929 18:38:17.896780 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":123,"threads_started":1}
I20260929 18:38:17.897807 23505 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P adc2adc306ab41da812fad0232231889: No bootstrap required, opened a new log
I20260929 18:38:17.901574 23505 raft_consensus.cc:348] T 00000000000000000000000000000000 P adc2adc306ab41da812fad0232231889 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adc2adc306ab41da812fad0232231889" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 43117 } }
I20260929 18:38:17.901779 23505 raft_consensus.cc:374] T 00000000000000000000000000000000 P adc2adc306ab41da812fad0232231889 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:38:17.901847 23505 raft_consensus.cc:729] T 00000000000000000000000000000000 P adc2adc306ab41da812fad0232231889 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: adc2adc306ab41da812fad0232231889, State: Initialized, Role: FOLLOWER
I20260929 18:38:17.902290 23505 consensus_queue.cc:260] T 00000000000000000000000000000000 P adc2adc306ab41da812fad0232231889 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adc2adc306ab41da812fad0232231889" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 43117 } }
I20260929 18:38:17.902475 23510 raft_consensus.cc:482] T 00000000000000000000000000000000 P adc2adc306ab41da812fad0232231889 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:38:17.902638 23510 raft_consensus.cc:504] T 00000000000000000000000000000000 P adc2adc306ab41da812fad0232231889 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adc2adc306ab41da812fad0232231889" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 43117 } }
I20260929 18:38:17.902455 23505 raft_consensus.cc:388] T 00000000000000000000000000000000 P adc2adc306ab41da812fad0232231889 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:38:17.902971 23510 leader_election.cc:302] T 00000000000000000000000000000000 P adc2adc306ab41da812fad0232231889 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: adc2adc306ab41da812fad0232231889; no voters:
I20260929 18:38:17.903069 23505 raft_consensus.cc:482] T 00000000000000000000000000000000 P adc2adc306ab41da812fad0232231889 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:38:17.903164 23505 raft_consensus.cc:3037] T 00000000000000000000000000000000 P adc2adc306ab41da812fad0232231889 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:38:17.903322 23510 leader_election.cc:288] T 00000000000000000000000000000000 P adc2adc306ab41da812fad0232231889 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers
I20260929 18:38:17.904147 23505 raft_consensus.cc:504] T 00000000000000000000000000000000 P adc2adc306ab41da812fad0232231889 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adc2adc306ab41da812fad0232231889" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 43117 } }
I20260929 18:38:17.904291 23505 leader_election.cc:302] T 00000000000000000000000000000000 P adc2adc306ab41da812fad0232231889 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: adc2adc306ab41da812fad0232231889; no voters:
I20260929 18:38:17.904376 23505 leader_election.cc:288] T 00000000000000000000000000000000 P adc2adc306ab41da812fad0232231889 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:38:17.904385 23511 raft_consensus.cc:2741] T 00000000000000000000000000000000 P adc2adc306ab41da812fad0232231889 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260929 18:38:17.904845 23512 sys_catalog.cc:455] T 00000000000000000000000000000000 P adc2adc306ab41da812fad0232231889 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adc2adc306ab41da812fad0232231889" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 43117 } } }
I20260929 18:38:17.904978 23512 sys_catalog.cc:458] T 00000000000000000000000000000000 P adc2adc306ab41da812fad0232231889 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:38:17.905139 23510 raft_consensus.cc:2781] T 00000000000000000000000000000000 P adc2adc306ab41da812fad0232231889 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:38:17.905257 23510 raft_consensus.cc:686] T 00000000000000000000000000000000 P adc2adc306ab41da812fad0232231889 [term 1 LEADER]: Becoming Leader. State: Replica: adc2adc306ab41da812fad0232231889, State: Running, Role: LEADER
I20260929 18:38:17.905515 23510 consensus_queue.cc:237] T 00000000000000000000000000000000 P adc2adc306ab41da812fad0232231889 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adc2adc306ab41da812fad0232231889" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 43117 } }
I20260929 18:38:17.905612 23505 sys_catalog.cc:564] T 00000000000000000000000000000000 P adc2adc306ab41da812fad0232231889 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:38:17.906716 23511 sys_catalog.cc:455] T 00000000000000000000000000000000 P adc2adc306ab41da812fad0232231889 [sys.catalog]: SysCatalogTable state changed. Reason: New leader adc2adc306ab41da812fad0232231889. Latest consensus state: current_term: 1 leader_uuid: "adc2adc306ab41da812fad0232231889" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adc2adc306ab41da812fad0232231889" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 43117 } } }
I20260929 18:38:17.906821 23511 sys_catalog.cc:458] T 00000000000000000000000000000000 P adc2adc306ab41da812fad0232231889 [sys.catalog]: This master's current role is: LEADER
I20260929 18:38:17.907194 23519 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:38:17.910272 23519 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:38:17.915928 23519 catalog_manager.cc:1347] Generated new cluster ID: fc4e704594e040b980f90a778fdcc562
I20260929 18:38:17.915994 23519 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:38:17.927949 23519 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:38:17.928311 23519 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:38:17.935724 23519 catalog_manager.cc:5868] T 00000000000000000000000000000000 P adc2adc306ab41da812fad0232231889: Generated new TSK 0
I20260929 18:38:17.936401 23519 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:38:17.942771 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707036041078-13712-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707036041078-13712-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707036041078-13712-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707036041078-13712-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.100.1:0
--local_ip_for_outbound_sockets=127.13.100.1
--webserver_interface=127.13.100.1
--webserver_port=0
--tserver_master_addrs=127.13.100.62:43117
--builtin_ntp_servers=127.13.100.20:39109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260929 18:38:18.061084 23529 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:38:18.061342 23529 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:38:18.061383 23529 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:38:18.065390 23529 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:38:18.065444 23529 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:38:18.065500 23529 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.100.1
I20260929 18:38:18.070297 23529 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.100.20:39109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707036041078-13712-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707036041078-13712-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.100.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707036041078-13712-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.100.1
--webserver_port=0
--tserver_master_addrs=127.13.100.62:43117
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23529
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.100.1
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707036041078-13712-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:38:18.071324 23529 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:38:18.072615 23529 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:38:18.082216 23535 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:38:18.082345 23529 server_base.cc:1034] running on GCE node
W20260929 18:38:18.082260 23537 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:38:18.082311 23534 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:38:18.082868 23529 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:38:18.083472 23529 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:38:18.084661 23529 hybrid_clock.cc:648] HybridClock initialized: now 1790707098084645 us; error 32 us; skew 500 ppm
I20260929 18:38:18.086506 23529 webserver.cc:460] Webserver started at http://127.13.100.1:35007/ using document root <none> and password file <none>
I20260929 18:38:18.087067 23529 fs_manager.cc:359] Metadata directory not provided
I20260929 18:38:18.087157 23529 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:38:18.087378 23529 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:38:18.089416 23529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707036041078-13712-0/minicluster-data/ts-0/data/instance:
uuid: "45db16231ad34557a836f6e41d7e843c"
format_stamp: "Formatted at 2026-09-29 18:38:18 on dist-test-slave-j1xs"
server_key: "1d0f5ee888965e30c2fc7c4f6231b588"
server_key_iv: "a5266cbbbf6f8d6a7f2b0356f280b993"
server_key_version: "encryptionkey@0"
I20260929 18:38:18.090049 23529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707036041078-13712-0/minicluster-data/ts-0/wal/instance:
uuid: "45db16231ad34557a836f6e41d7e843c"
format_stamp: "Formatted at 2026-09-29 18:38:18 on dist-test-slave-j1xs"
server_key: "1d0f5ee888965e30c2fc7c4f6231b588"
server_key_iv: "a5266cbbbf6f8d6a7f2b0356f280b993"
server_key_version: "encryptionkey@0"
I20260929 18:38:18.093653 23529 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:38:18.096379 23543 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:38:18.097537 23529 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:38:18.097674 23529 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707036041078-13712-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707036041078-13712-0/minicluster-data/ts-0/wal
uuid: "45db16231ad34557a836f6e41d7e843c"
format_stamp: "Formatted at 2026-09-29 18:38:18 on dist-test-slave-j1xs"
server_key: "1d0f5ee888965e30c2fc7c4f6231b588"
server_key_iv: "a5266cbbbf6f8d6a7f2b0356f280b993"
server_key_version: "encryptionkey@0"
I20260929 18:38:18.097787 23529 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707036041078-13712-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707036041078-13712-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707036041078-13712-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:38:18.102262 23529 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:38:18.102828 23529 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:38:18.103012 23529 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:38:18.103528 23529 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:38:18.104362 23529 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:38:18.104434 23529 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:38:18.104499 23529 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:38:18.104533 23529 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:38:18.113142 23529 rpc_server.cc:307] RPC server started. Bound to: 127.13.100.1:38229
I20260929 18:38:18.113178 23656 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.100.1:38229 every 8 connection(s)
I20260929 18:38:18.114003 23529 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707036041078-13712-0/minicluster-data/ts-0/data/info.pb
I20260929 18:38:18.119684 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 23529
I20260929 18:38:18.119700 23508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:18.115651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.1:39507 (local address 127.13.100.62:43117)
0929 18:38:18.115783 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 18:38:18.115787 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:38:18.116703 (+ 916us) server_negotiation.cc:407] Connection header received
0929 18:38:18.118020 (+ 1317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:38:18.118023 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:38:18.118066 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:38:18.118185 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:38:18.119245 (+ 1060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:38:18.119249 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:38:18.119252 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:38:18.119264 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:38:18.119284 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:38:18.119293 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:38:18.119392 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0929 18:38:18.119532 (+ 140us) server_negotiation.cc:299] Negotiation successful
0929 18:38:18.119574 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 18:38:18.119817 13712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707036041078-13712-0/minicluster-data/ts-0/wal/instance
I20260929 18:38:18.120086 13712 external_mini_cluster.cc:1428] Setting key 372574c2a2bc741ae8d65665481b9fa2
I20260929 18:38:18.120594 23659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:18.115964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:43117 (local address 127.13.100.1:39507)
0929 18:38:18.116550 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:18.116587 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 18:38:18.117778 (+ 1191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:38:18.118334 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:38:18.118339 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:38:18.118418 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:38:18.118999 (+ 581us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:38:18.119001 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:38:18.119037 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:38:18.119100 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:38:18.119385 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:38:18.119393 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:38:18.119400 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 18:38:18.119569 (+ 169us) client_negotiation.cc:240] Negotiation successful
0929 18:38:18.119858 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":96,"threads_started":1}
I20260929 18:38:18.121716 23657 heartbeater.cc:347] Connected to a master server at 127.13.100.62:43117
I20260929 18:38:18.121963 23657 heartbeater.cc:464] Registering TS with master...
I20260929 18:38:18.121946 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707036041078-13712-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707036041078-13712-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707036041078-13712-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707036041078-13712-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.100.2:0
--local_ip_for_outbound_sockets=127.13.100.2
--webserver_interface=127.13.100.2
--webserver_port=0
--tserver_master_addrs=127.13.100.62:43117
--builtin_ntp_servers=127.13.100.20:39109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260929 18:38:18.122354 23657 heartbeater.cc:511] Master 127.13.100.62:43117 requested a full tablet report, sending...
I20260929 18:38:18.123699 23468 ts_manager.cc:194] Registered new tserver with Master: 45db16231ad34557a836f6e41d7e843c (127.13.100.1:38229)
W20260929 18:38:18.229871 23660 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:38:18.230125 23660 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:38:18.230162 23660 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:38:18.234259 23660 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:38:18.234305 23660 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:38:18.234393 23660 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.100.2
I20260929 18:38:18.239061 23660 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.100.20:39109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707036041078-13712-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707036041078-13712-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.100.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707036041078-13712-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.100.2
--webserver_port=0
--tserver_master_addrs=127.13.100.62:43117
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23660
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.100.2
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707036041078-13712-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:38:18.240033 23660 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:38:18.241322 23660 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:38:18.250731 23666 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:38:18.250738 23668 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:38:18.250756 23665 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:38:18.250857 23660 server_base.cc:1034] running on GCE node
I20260929 18:38:18.251331 23660 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:38:18.252049 23660 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:38:18.253222 23660 hybrid_clock.cc:648] HybridClock initialized: now 1790707098253196 us; error 46 us; skew 500 ppm
I20260929 18:38:18.255016 23660 webserver.cc:460] Webserver started at http://127.13.100.2:33611/ using document root <none> and password file <none>
I20260929 18:38:18.255575 23660 fs_manager.cc:359] Metadata directory not provided
I20260929 18:38:18.255671 23660 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:38:18.255880 23660 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:38:18.257974 23660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707036041078-13712-0/minicluster-data/ts-1/data/instance:
uuid: "3c22d4bbdb4946349420b6a7b48a1231"
format_stamp: "Formatted at 2026-09-29 18:38:18 on dist-test-slave-j1xs"
server_key: "eab5f3e44814d16a35179fdbe6d29dbf"
server_key_iv: "738a8b3f99851e2882064935d3db44c8"
server_key_version: "encryptionkey@0"
I20260929 18:38:18.258549 23660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707036041078-13712-0/minicluster-data/ts-1/wal/instance:
uuid: "3c22d4bbdb4946349420b6a7b48a1231"
format_stamp: "Formatted at 2026-09-29 18:38:18 on dist-test-slave-j1xs"
server_key: "eab5f3e44814d16a35179fdbe6d29dbf"
server_key_iv: "738a8b3f99851e2882064935d3db44c8"
server_key_version: "encryptionkey@0"
I20260929 18:38:18.262102 23660 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260929 18:38:18.264572 23674 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:38:18.266386 23660 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:38:18.266521 23660 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707036041078-13712-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707036041078-13712-0/minicluster-data/ts-1/wal
uuid: "3c22d4bbdb4946349420b6a7b48a1231"
format_stamp: "Formatted at 2026-09-29 18:38:18 on dist-test-slave-j1xs"
server_key: "eab5f3e44814d16a35179fdbe6d29dbf"
server_key_iv: "738a8b3f99851e2882064935d3db44c8"
server_key_version: "encryptionkey@0"
I20260929 18:38:18.266637 23660 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707036041078-13712-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707036041078-13712-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707036041078-13712-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:38:18.271122 23660 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:38:18.271685 23660 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:38:18.271865 23660 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:38:18.272367 23660 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:38:18.273214 23660 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:38:18.273283 23660 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:38:18.273350 23660 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:38:18.273399 23660 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:38:18.282137 23660 rpc_server.cc:307] RPC server started. Bound to: 127.13.100.2:44461
I20260929 18:38:18.282176 23787 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.100.2:44461 every 8 connection(s)
I20260929 18:38:18.282948 23660 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707036041078-13712-0/minicluster-data/ts-1/data/info.pb
I20260929 18:38:18.287504 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 23660
I20260929 18:38:18.287640 13712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707036041078-13712-0/minicluster-data/ts-1/wal/instance
I20260929 18:38:18.287922 13712 external_mini_cluster.cc:1428] Setting key c09fd9ce623efb401f3db5f1ccf8b795
I20260929 18:38:18.288928 23508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:18.284612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.2:49489 (local address 127.13.100.62:43117)
0929 18:38:18.284738 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0929 18:38:18.284741 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:38:18.285713 (+ 972us) server_negotiation.cc:407] Connection header received
0929 18:38:18.287146 (+ 1433us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:38:18.287149 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:38:18.287192 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:38:18.287297 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:38:18.288461 (+ 1164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:38:18.288465 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:38:18.288467 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:38:18.288478 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:38:18.288506 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:38:18.288517 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:38:18.288610 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 18:38:18.288760 (+ 150us) server_negotiation.cc:299] Negotiation successful
0929 18:38:18.288803 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 18:38:18.289759 23790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:18.284976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:43117 (local address 127.13.100.2:49489)
0929 18:38:18.285581 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:18.285617 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 18:38:18.286916 (+ 1299us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:38:18.287450 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:38:18.287456 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:38:18.287539 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:38:18.288206 (+ 667us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:38:18.288215 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:38:18.288251 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:38:18.288313 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:38:18.288608 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:38:18.288618 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:38:18.288625 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 18:38:18.288800 (+ 175us) client_negotiation.cc:240] Negotiation successful
0929 18:38:18.289028 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":98,"threads_started":1}
I20260929 18:38:18.289824 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707036041078-13712-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707036041078-13712-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707036041078-13712-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707036041078-13712-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.100.3:0
--local_ip_for_outbound_sockets=127.13.100.3
--webserver_interface=127.13.100.3
--webserver_port=0
--tserver_master_addrs=127.13.100.62:43117
--builtin_ntp_servers=127.13.100.20:39109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260929 18:38:18.290844 23788 heartbeater.cc:347] Connected to a master server at 127.13.100.62:43117
I20260929 18:38:18.291092 23788 heartbeater.cc:464] Registering TS with master...
I20260929 18:38:18.291503 23788 heartbeater.cc:511] Master 127.13.100.62:43117 requested a full tablet report, sending...
I20260929 18:38:18.292451 23468 ts_manager.cc:194] Registered new tserver with Master: 3c22d4bbdb4946349420b6a7b48a1231 (127.13.100.2:44461)
W20260929 18:38:18.397930 23791 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:38:18.398229 23791 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:38:18.398272 23791 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:38:18.402570 23791 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:38:18.402654 23791 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:38:18.402745 23791 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.100.3
I20260929 18:38:18.408134 23791 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.100.20:39109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707036041078-13712-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707036041078-13712-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.100.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707036041078-13712-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.100.3
--webserver_port=0
--tserver_master_addrs=127.13.100.62:43117
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23791
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.100.3
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707036041078-13712-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:38:18.409224 23791 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:38:18.410645 23791 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:38:18.419960 23797 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:38:18.419932 23799 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:38:18.420133 23791 server_base.cc:1034] running on GCE node
W20260929 18:38:18.420012 23796 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:38:18.420634 23791 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:38:18.421272 23791 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:38:18.422461 23791 hybrid_clock.cc:648] HybridClock initialized: now 1790707098422451 us; error 30 us; skew 500 ppm
I20260929 18:38:18.424270 23791 webserver.cc:460] Webserver started at http://127.13.100.3:42353/ using document root <none> and password file <none>
I20260929 18:38:18.424876 23791 fs_manager.cc:359] Metadata directory not provided
I20260929 18:38:18.424978 23791 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:38:18.425242 23791 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:38:18.427387 23791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707036041078-13712-0/minicluster-data/ts-2/data/instance:
uuid: "321c039a4d75416d8c701cf89baa21d9"
format_stamp: "Formatted at 2026-09-29 18:38:18 on dist-test-slave-j1xs"
server_key: "0d42d3598e7ba3789c82381a486c3aa0"
server_key_iv: "83b02284aa5a83f768345e82a20d88ab"
server_key_version: "encryptionkey@0"
I20260929 18:38:18.427956 23791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707036041078-13712-0/minicluster-data/ts-2/wal/instance:
uuid: "321c039a4d75416d8c701cf89baa21d9"
format_stamp: "Formatted at 2026-09-29 18:38:18 on dist-test-slave-j1xs"
server_key: "0d42d3598e7ba3789c82381a486c3aa0"
server_key_iv: "83b02284aa5a83f768345e82a20d88ab"
server_key_version: "encryptionkey@0"
I20260929 18:38:18.431721 23791 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 18:38:18.434453 23805 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:38:18.435963 23791 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:38:18.436116 23791 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707036041078-13712-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707036041078-13712-0/minicluster-data/ts-2/wal
uuid: "321c039a4d75416d8c701cf89baa21d9"
format_stamp: "Formatted at 2026-09-29 18:38:18 on dist-test-slave-j1xs"
server_key: "0d42d3598e7ba3789c82381a486c3aa0"
server_key_iv: "83b02284aa5a83f768345e82a20d88ab"
server_key_version: "encryptionkey@0"
I20260929 18:38:18.436225 23791 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707036041078-13712-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707036041078-13712-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707036041078-13712-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:38:18.440783 23791 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:38:18.441372 23791 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:38:18.441550 23791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:38:18.442137 23791 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:38:18.442998 23791 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:38:18.443042 23791 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:38:18.443109 23791 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:38:18.443156 23791 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:38:18.451683 23791 rpc_server.cc:307] RPC server started. Bound to: 127.13.100.3:32903
I20260929 18:38:18.451742 23918 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.100.3:32903 every 8 connection(s)
I20260929 18:38:18.452611 23791 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707036041078-13712-0/minicluster-data/ts-2/data/info.pb
I20260929 18:38:18.456176 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 23791
I20260929 18:38:18.456274 13712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707036041078-13712-0/minicluster-data/ts-2/wal/instance
I20260929 18:38:18.456616 13712 external_mini_cluster.cc:1428] Setting key 2768f973a4518952b6a812306246108a
I20260929 18:38:18.458959 23508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:18.454364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.3:37827 (local address 127.13.100.62:43117)
0929 18:38:18.454489 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0929 18:38:18.454492 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:38:18.455545 (+ 1053us) server_negotiation.cc:407] Connection header received
0929 18:38:18.457171 (+ 1626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:38:18.457174 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:38:18.457216 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:38:18.457320 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:38:18.458512 (+ 1192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:38:18.458515 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:38:18.458518 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:38:18.458529 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:38:18.458553 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:38:18.458568 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:38:18.458656 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0929 18:38:18.458798 (+ 142us) server_negotiation.cc:299] Negotiation successful
0929 18:38:18.458839 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 18:38:18.459803 23921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:18.454761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:43117 (local address 127.13.100.3:37827)
0929 18:38:18.455415 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:18.455453 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 18:38:18.456976 (+ 1523us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:38:18.457470 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:38:18.457476 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:38:18.457593 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:38:18.458261 (+ 668us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:38:18.458263 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:38:18.458301 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:38:18.458366 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:38:18.458648 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:38:18.458655 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:38:18.458662 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 18:38:18.458839 (+ 177us) client_negotiation.cc:240] Negotiation successful
0929 18:38:18.459063 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":100,"threads_started":1}
I20260929 18:38:18.460956 23919 heartbeater.cc:347] Connected to a master server at 127.13.100.62:43117
I20260929 18:38:18.461164 23919 heartbeater.cc:464] Registering TS with master...
I20260929 18:38:18.461522 23919 heartbeater.cc:511] Master 127.13.100.62:43117 requested a full tablet report, sending...
I20260929 18:38:18.462478 23468 ts_manager.cc:194] Registered new tserver with Master: 321c039a4d75416d8c701cf89baa21d9 (127.13.100.3:32903)
I20260929 18:38:18.470549 13712 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260929 18:38:18.473408 23508 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:38:18.472561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49768 (local address 127.13.100.62:43117)
0929 18:38:18.472693 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 18:38:18.472697 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:38:18.472844 (+ 147us) server_negotiation.cc:407] Connection header received
0929 18:38:18.472956 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:38:18.472958 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:38:18.473011 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:38:18.473081 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:38:18.473265 (+ 184us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49768: BlockingRecv error: recv got EOF from 127.0.0.1:49768 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 18:38:18.473696 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 23529
I20260929 18:38:18.479436 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 23660
I20260929 18:38:18.484876 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 23791
I20260929 18:38:18.489863 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 23438
2026-09-29T18:38:18Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (792 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707036041078-13712-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:38:18 dist-test-slave-j1xs krb5kdc[23929](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:38:18 dist-test-slave-j1xs krb5kdc[23929](info): set up 2 sockets
Sep 29 18:38:18 dist-test-slave-j1xs krb5kdc[23929](info): commencing operation
krb5kdc: starting...
W20260929 18:38:19.538910 13712 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707036041078-13712-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707036041078-13712-0/krb5kdc/test-user.keytab.
Sep 29 18:38:19 dist-test-slave-j1xs krb5kdc[23929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707099, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:38:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:38:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.100.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.100.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.100.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.62.keytab.
Entry for principal kudu/127.13.100.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.62.keytab.
I20260929 18:38:19.666448 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707036041078-13712-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707036041078-13712-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707036041078-13712-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.100.62:33821
--webserver_interface=127.13.100.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.62.keytab
--principal=kudu/127.13.100.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.100.20:42683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.100.62:33821
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707036041078-13712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707036041078-13712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707036041078-13712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:38:19.777227 23942 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:38:19.777486 23942 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:38:19.777527 23942 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:38:19.781484 23942 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:38:19.781543 23942 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:38:19.781562 23942 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:38:19.781579 23942 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:38:19.781594 23942 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:38:19.786962 23942 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.100.20:42683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707036041078-13712-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707036041078-13712-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.100.62:33821
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.62.keytab
--principal=kudu/127.13.100.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.100.62:33821
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.100.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23942
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707036041078-13712-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:38:19.787920 23942 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:38:19.788887 23942 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:38:19.797353 23948 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:38:19.797474 23942 server_base.cc:1034] running on GCE node
W20260929 18:38:19.797382 23950 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:38:19.797379 23947 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:38:19.798192 23942 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:38:19.799211 23942 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:38:19.800422 23942 hybrid_clock.cc:648] HybridClock initialized: now 1790707099800392 us; error 48 us; skew 500 ppm
Sep 29 18:38:19 dist-test-slave-j1xs krb5kdc[23929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707099, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:38:19.803185 23942 init.cc:377] Logged in from keytab as kudu/127.13.100.62@KRBTEST.COM (short username kudu)
I20260929 18:38:19.804049 23942 webserver.cc:460] Webserver started at http://127.13.100.62:34747/ using document root <none> and password file <none>
I20260929 18:38:19.804584 23942 fs_manager.cc:359] Metadata directory not provided
I20260929 18:38:19.804673 23942 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:38:19.804891 23942 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:38:19.806923 23942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707036041078-13712-0/minicluster-data/master-0/data/instance:
uuid: "c7182844bcef446998295760fc1bb3cd"
format_stamp: "Formatted at 2026-09-29 18:38:19 on dist-test-slave-j1xs"
server_key: "09dc8bc1a4836b2db4d1247bdf85e7a6"
server_key_iv: "630389d450df9532a316a077a771a298"
server_key_version: "encryptionkey@0"
I20260929 18:38:19.807475 23942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707036041078-13712-0/minicluster-data/master-0/wal/instance:
uuid: "c7182844bcef446998295760fc1bb3cd"
format_stamp: "Formatted at 2026-09-29 18:38:19 on dist-test-slave-j1xs"
server_key: "09dc8bc1a4836b2db4d1247bdf85e7a6"
server_key_iv: "630389d450df9532a316a077a771a298"
server_key_version: "encryptionkey@0"
I20260929 18:38:19.811250 23942 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 18:38:19.813663 23957 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:38:19.815052 23942 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:38:19.815202 23942 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707036041078-13712-0/minicluster-data/master-0/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707036041078-13712-0/minicluster-data/master-0/wal
uuid: "c7182844bcef446998295760fc1bb3cd"
format_stamp: "Formatted at 2026-09-29 18:38:19 on dist-test-slave-j1xs"
server_key: "09dc8bc1a4836b2db4d1247bdf85e7a6"
server_key_iv: "630389d450df9532a316a077a771a298"
server_key_version: "encryptionkey@0"
I20260929 18:38:19.815320 23942 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707036041078-13712-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707036041078-13712-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707036041078-13712-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:38:19.832645 23942 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:38:19.835919 23942 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:38:19.836113 23942 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:38:19.843101 23942 rpc_server.cc:307] RPC server started. Bound to: 127.13.100.62:33821
I20260929 18:38:19.843384 24009 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.100.62:33821 every 8 connection(s)
I20260929 18:38:19.844043 23942 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
I20260929 18:38:19.846692 24010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:38:19.851985 24010 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7182844bcef446998295760fc1bb3cd: Bootstrap starting.
I20260929 18:38:19.853039 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 23942
I20260929 18:38:19.853183 13712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707036041078-13712-0/minicluster-data/master-0/wal/instance
I20260929 18:38:19.853459 13712 external_mini_cluster.cc:1428] Setting key 23f6a1eb8ea941079efb0e51f5afcd8c
I20260929 18:38:19.855176 24010 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c7182844bcef446998295760fc1bb3cd: Neither blocks nor log segments found. Creating new log.
I20260929 18:38:19.855896 24010 log.cc:824] T 00000000000000000000000000000000 P c7182844bcef446998295760fc1bb3cd: Log is configured to *not* fsync() on all Append() calls
I20260929 18:38:19.858315 24010 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7182844bcef446998295760fc1bb3cd: No bootstrap required, opened a new log
I20260929 18:38:19.862957 24010 raft_consensus.cc:348] T 00000000000000000000000000000000 P c7182844bcef446998295760fc1bb3cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7182844bcef446998295760fc1bb3cd" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 33821 } }
Sep 29 18:38:19 dist-test-slave-j1xs krb5kdc[23929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707099, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.100.62@KRBTEST.COM
I20260929 18:38:19.863216 24010 raft_consensus.cc:374] T 00000000000000000000000000000000 P c7182844bcef446998295760fc1bb3cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:38:19.863296 24010 raft_consensus.cc:729] T 00000000000000000000000000000000 P c7182844bcef446998295760fc1bb3cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7182844bcef446998295760fc1bb3cd, State: Initialized, Role: FOLLOWER
I20260929 18:38:19.863730 24010 consensus_queue.cc:260] T 00000000000000000000000000000000 P c7182844bcef446998295760fc1bb3cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7182844bcef446998295760fc1bb3cd" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 33821 } }
I20260929 18:38:19.863875 24010 raft_consensus.cc:388] T 00000000000000000000000000000000 P c7182844bcef446998295760fc1bb3cd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:38:19.863940 24010 raft_consensus.cc:482] T 00000000000000000000000000000000 P c7182844bcef446998295760fc1bb3cd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:38:19.864032 24010 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c7182844bcef446998295760fc1bb3cd [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:38:19.865228 24010 raft_consensus.cc:504] T 00000000000000000000000000000000 P c7182844bcef446998295760fc1bb3cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7182844bcef446998295760fc1bb3cd" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 33821 } }
I20260929 18:38:19.865572 24010 leader_election.cc:302] T 00000000000000000000000000000000 P c7182844bcef446998295760fc1bb3cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c7182844bcef446998295760fc1bb3cd; no voters:
I20260929 18:38:19.865921 24010 leader_election.cc:288] T 00000000000000000000000000000000 P c7182844bcef446998295760fc1bb3cd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:38:19.865993 24015 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c7182844bcef446998295760fc1bb3cd [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:38:19.866200 24015 raft_consensus.cc:686] T 00000000000000000000000000000000 P c7182844bcef446998295760fc1bb3cd [term 1 LEADER]: Becoming Leader. State: Replica: c7182844bcef446998295760fc1bb3cd, State: Running, Role: LEADER
I20260929 18:38:19.866540 24015 consensus_queue.cc:237] T 00000000000000000000000000000000 P c7182844bcef446998295760fc1bb3cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7182844bcef446998295760fc1bb3cd" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 33821 } }
I20260929 18:38:19.867218 24010 sys_catalog.cc:564] T 00000000000000000000000000000000 P c7182844bcef446998295760fc1bb3cd [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:38:19.868207 24016 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7182844bcef446998295760fc1bb3cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c7182844bcef446998295760fc1bb3cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7182844bcef446998295760fc1bb3cd" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 33821 } } }
I20260929 18:38:19.868332 24016 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7182844bcef446998295760fc1bb3cd [sys.catalog]: This master's current role is: LEADER
I20260929 18:38:19.868707 24017 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7182844bcef446998295760fc1bb3cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader c7182844bcef446998295760fc1bb3cd. Latest consensus state: current_term: 1 leader_uuid: "c7182844bcef446998295760fc1bb3cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7182844bcef446998295760fc1bb3cd" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 33821 } } }
I20260929 18:38:19.868868 24017 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7182844bcef446998295760fc1bb3cd [sys.catalog]: This master's current role is: LEADER
I20260929 18:38:19.870131 24013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:19.854631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42790 (local address 127.13.100.62:33821)
0929 18:38:19.855053 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0929 18:38:19.855067 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 18:38:19.855101 (+ 34us) server_negotiation.cc:407] Connection header received
0929 18:38:19.855900 (+ 799us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:38:19.855929 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:38:19.856438 (+ 509us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:38:19.856835 (+ 397us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:38:19.857660 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:19.858965 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:19.860281 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:19.860777 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:19.860897 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:19.863967 (+ 3070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:38:19.864025 (+ 58us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:38:19.864041 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:38:19.864072 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:38:19.866258 (+ 2186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:38:19.866792 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:38:19.866804 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:38:19.866809 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:38:19.866915 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:38:19.867237 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:38:19.867240 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:38:19.867242 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:38:19.867774 (+ 532us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:38:19.868662 (+ 888us) server_negotiation.cc:1035] Waiting for connection context
0929 18:38:19.868807 (+ 145us) server_negotiation.cc:299] Negotiation successful
0929 18:38:19.869117 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":131,"threads_started":1}
W20260929 18:38:19.872323 24031 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c7182844bcef446998295760fc1bb3cd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:38:19.872396 24031 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:38:19.872457 24024 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:38:19.873106 24024 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:38:19.878147 24024 catalog_manager.cc:1347] Generated new cluster ID: f32c30382fae4efcaad94c3c19f45ebf
I20260929 18:38:19.878216 24024 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:38:19.894405 24024 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:38:19.895346 24024 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:38:19.904738 24024 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c7182844bcef446998295760fc1bb3cd: Generated new TSK 0
I20260929 18:38:19.905393 24024 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.100.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.100.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.100.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.1.keytab.
Entry for principal kudu/127.13.100.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.1.keytab.
I20260929 18:38:19.937405 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707036041078-13712-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707036041078-13712-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707036041078-13712-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707036041078-13712-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.100.1:0
--local_ip_for_outbound_sockets=127.13.100.1
--webserver_interface=127.13.100.1
--webserver_port=0
--tserver_master_addrs=127.13.100.62:33821
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.1.keytab
--principal=kudu/127.13.100.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.100.20:42683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707036041078-13712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707036041078-13712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707036041078-13712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:38:20.067061 24036 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:38:20.067327 24036 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:38:20.067411 24036 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:38:20.071564 24036 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:38:20.071627 24036 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:38:20.071686 24036 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.100.1
I20260929 18:38:20.076781 24036 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.100.20:42683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707036041078-13712-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707036041078-13712-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.1.keytab
--principal=kudu/127.13.100.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.100.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707036041078-13712-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.100.1
--webserver_port=0
--tserver_master_addrs=127.13.100.62:33821
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24036
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.100.1
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707036041078-13712-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:38:20.077725 24036 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:38:20.078728 24036 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:38:20.087984 24042 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:38:20.087963 24044 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:38:20.087963 24041 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:38:20.088706 24036 server_base.cc:1034] running on GCE node
I20260929 18:38:20.089085 24036 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:38:20.089658 24036 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:38:20.090868 24036 hybrid_clock.cc:648] HybridClock initialized: now 1790707100090853 us; error 64 us; skew 500 ppm
Sep 29 18:38:20 dist-test-slave-j1xs krb5kdc[23929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707100, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:38:20.093573 24036 init.cc:377] Logged in from keytab as kudu/127.13.100.1@KRBTEST.COM (short username kudu)
I20260929 18:38:20.094499 24036 webserver.cc:460] Webserver started at http://127.13.100.1:36643/ using document root <none> and password file <none>
I20260929 18:38:20.095009 24036 fs_manager.cc:359] Metadata directory not provided
I20260929 18:38:20.095067 24036 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:38:20.095232 24036 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:38:20.097179 24036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707036041078-13712-0/minicluster-data/ts-0/data/instance:
uuid: "7f3ea37b5e0945599a09cc5f1e352e49"
format_stamp: "Formatted at 2026-09-29 18:38:20 on dist-test-slave-j1xs"
server_key: "b971a155d0375ff9d9def8c306a8efdd"
server_key_iv: "556d0e6ec6fd28dbf68b073c4dbdf083"
server_key_version: "encryptionkey@0"
I20260929 18:38:20.097675 24036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707036041078-13712-0/minicluster-data/ts-0/wal/instance:
uuid: "7f3ea37b5e0945599a09cc5f1e352e49"
format_stamp: "Formatted at 2026-09-29 18:38:20 on dist-test-slave-j1xs"
server_key: "b971a155d0375ff9d9def8c306a8efdd"
server_key_iv: "556d0e6ec6fd28dbf68b073c4dbdf083"
server_key_version: "encryptionkey@0"
I20260929 18:38:20.101325 24036 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260929 18:38:20.103708 24051 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:38:20.104895 24036 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260929 18:38:20.105026 24036 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707036041078-13712-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707036041078-13712-0/minicluster-data/ts-0/wal
uuid: "7f3ea37b5e0945599a09cc5f1e352e49"
format_stamp: "Formatted at 2026-09-29 18:38:20 on dist-test-slave-j1xs"
server_key: "b971a155d0375ff9d9def8c306a8efdd"
server_key_iv: "556d0e6ec6fd28dbf68b073c4dbdf083"
server_key_version: "encryptionkey@0"
I20260929 18:38:20.105132 24036 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707036041078-13712-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707036041078-13712-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707036041078-13712-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:38:20.115211 24036 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:38:20.117862 24036 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:38:20.118145 24036 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:38:20.118788 24036 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:38:20.119661 24036 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:38:20.119714 24036 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:38:20.119778 24036 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:38:20.119809 24036 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:38:20.129232 24036 rpc_server.cc:307] RPC server started. Bound to: 127.13.100.1:38125
I20260929 18:38:20.129264 24164 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.100.1:38125 every 8 connection(s)
I20260929 18:38:20.130204 24036 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707036041078-13712-0/minicluster-data/ts-0/data/info.pb
I20260929 18:38:20.134572 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 24036
I20260929 18:38:20.134725 13712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707036041078-13712-0/minicluster-data/ts-0/wal/instance
I20260929 18:38:20.135030 13712 external_mini_cluster.cc:1428] Setting key 935b8b7ffa1d75d3f3f4d2e92c82c5f7
Sep 29 18:38:20 dist-test-slave-j1xs krb5kdc[23929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707100, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.1@KRBTEST.COM for kudu/127.13.100.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.13.100.2@KRBTEST.COM; defaulting to no policy
I20260929 18:38:20.146059 24013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:20.131965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.1:54589 (local address 127.13.100.62:33821)
0929 18:38:20.132127 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0929 18:38:20.132132 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:38:20.132953 (+ 821us) server_negotiation.cc:407] Connection header received
0929 18:38:20.134455 (+ 1502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:38:20.134458 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:38:20.134504 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:38:20.134614 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:38:20.136128 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:20.136920 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:20.138713 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:20.139116 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:20.139237 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:20.141987 (+ 2750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:38:20.141995 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:38:20.142000 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:38:20.142035 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:38:20.143811 (+ 1776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:38:20.144343 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:38:20.144347 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:38:20.144348 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:38:20.144441 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:38:20.144865 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:38:20.144869 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:38:20.144871 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:38:20.145110 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:38:20.145227 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0929 18:38:20.145765 (+ 538us) server_negotiation.cc:299] Negotiation successful
0929 18:38:20.145874 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 18:38:20.146927 24167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:20.132373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:33821 (local address 127.13.100.1:54589)
0929 18:38:20.132803 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:20.132844 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 18:38:20.134213 (+ 1369us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:38:20.134832 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:38:20.134842 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:38:20.135356 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:38:20.135968 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:20.135983 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:20.137056 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:20.137062 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:20.138586 (+ 1524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:20.138596 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:20.139326 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:20.139332 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:20.139559 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:20.140237 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:38:20.140254 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:38:20.141786 (+ 1532us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:38:20.143937 (+ 2151us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:38:20.143942 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:38:20.143955 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:38:20.144230 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:38:20.144610 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:38:20.144613 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:38:20.144614 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:38:20.144757 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:38:20.145249 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:38:20.145256 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:38:20.145520 (+ 264us) client_negotiation.cc:769] Sending connection context
0929 18:38:20.145698 (+ 178us) client_negotiation.cc:240] Negotiation successful
0929 18:38:20.146031 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":96,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.100.2@KRBTEST.COM" created.
I20260929 18:38:20.148298 24165 heartbeater.cc:347] Connected to a master server at 127.13.100.62:33821
I20260929 18:38:20.148590 24165 heartbeater.cc:464] Registering TS with master...
I20260929 18:38:20.149197 24165 heartbeater.cc:511] Master 127.13.100.62:33821 requested a full tablet report, sending...
I20260929 18:38:20.150887 23974 ts_manager.cc:194] Registered new tserver with Master: 7f3ea37b5e0945599a09cc5f1e352e49 (127.13.100.1:38125)
I20260929 18:38:20.151993 23974 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.100.1@KRBTEST.COM'} at 127.13.100.1:54589
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.100.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.2.keytab.
Entry for principal kudu/127.13.100.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.2.keytab.
I20260929 18:38:20.165294 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707036041078-13712-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707036041078-13712-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707036041078-13712-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707036041078-13712-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.100.2:0
--local_ip_for_outbound_sockets=127.13.100.2
--webserver_interface=127.13.100.2
--webserver_port=0
--tserver_master_addrs=127.13.100.62:33821
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.2.keytab
--principal=kudu/127.13.100.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.100.20:42683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707036041078-13712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707036041078-13712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707036041078-13712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:38:20.275714 24170 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:38:20.275974 24170 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:38:20.276017 24170 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:38:20.279930 24170 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:38:20.279990 24170 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:38:20.280048 24170 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.100.2
I20260929 18:38:20.285542 24170 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.100.20:42683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707036041078-13712-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707036041078-13712-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.2.keytab
--principal=kudu/127.13.100.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.100.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707036041078-13712-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.100.2
--webserver_port=0
--tserver_master_addrs=127.13.100.62:33821
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24170
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.100.2
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707036041078-13712-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:38:20.286584 24170 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:38:20.287544 24170 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:38:20.296942 24178 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:38:20.296938 24175 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:38:20.297035 24170 server_base.cc:1034] running on GCE node
W20260929 18:38:20.296942 24176 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:38:20.297587 24170 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:38:20.298322 24170 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:38:20.299511 24170 hybrid_clock.cc:648] HybridClock initialized: now 1790707100299493 us; error 32 us; skew 500 ppm
Sep 29 18:38:20 dist-test-slave-j1xs krb5kdc[23929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707100, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:38:20.302191 24170 init.cc:377] Logged in from keytab as kudu/127.13.100.2@KRBTEST.COM (short username kudu)
I20260929 18:38:20.302992 24170 webserver.cc:460] Webserver started at http://127.13.100.2:38361/ using document root <none> and password file <none>
I20260929 18:38:20.303498 24170 fs_manager.cc:359] Metadata directory not provided
I20260929 18:38:20.303555 24170 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:38:20.303719 24170 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:38:20.305647 24170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707036041078-13712-0/minicluster-data/ts-1/data/instance:
uuid: "077eb898bb964ae48b0cc63fe3422229"
format_stamp: "Formatted at 2026-09-29 18:38:20 on dist-test-slave-j1xs"
server_key: "2780c65fe47bfb09ec3665ae1b40d588"
server_key_iv: "83f62d0bde25e9ecd4189e06fa31d2d6"
server_key_version: "encryptionkey@0"
I20260929 18:38:20.306263 24170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707036041078-13712-0/minicluster-data/ts-1/wal/instance:
uuid: "077eb898bb964ae48b0cc63fe3422229"
format_stamp: "Formatted at 2026-09-29 18:38:20 on dist-test-slave-j1xs"
server_key: "2780c65fe47bfb09ec3665ae1b40d588"
server_key_iv: "83f62d0bde25e9ecd4189e06fa31d2d6"
server_key_version: "encryptionkey@0"
I20260929 18:38:20.310137 24170 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 18:38:20.312556 24185 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:38:20.313866 24170 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:38:20.314024 24170 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707036041078-13712-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707036041078-13712-0/minicluster-data/ts-1/wal
uuid: "077eb898bb964ae48b0cc63fe3422229"
format_stamp: "Formatted at 2026-09-29 18:38:20 on dist-test-slave-j1xs"
server_key: "2780c65fe47bfb09ec3665ae1b40d588"
server_key_iv: "83f62d0bde25e9ecd4189e06fa31d2d6"
server_key_version: "encryptionkey@0"
I20260929 18:38:20.314098 24170 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707036041078-13712-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707036041078-13712-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707036041078-13712-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:38:20.324856 24170 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:38:20.328810 24170 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:38:20.328969 24170 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:38:20.329487 24170 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:38:20.330448 24170 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:38:20.330503 24170 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:38:20.330543 24170 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:38:20.330559 24170 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:38:20.339340 24170 rpc_server.cc:307] RPC server started. Bound to: 127.13.100.2:40925
I20260929 18:38:20.339500 24298 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.100.2:40925 every 8 connection(s)
I20260929 18:38:20.340236 24170 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707036041078-13712-0/minicluster-data/ts-1/data/info.pb
I20260929 18:38:20.341845 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 24170
I20260929 18:38:20.342016 13712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707036041078-13712-0/minicluster-data/ts-1/wal/instance
I20260929 18:38:20.342327 13712 external_mini_cluster.cc:1428] Setting key 0daaec75ce51d123c61c4f84316affa2
Sep 29 18:38:20 dist-test-slave-j1xs krb5kdc[23929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707100, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.2@KRBTEST.COM for kudu/127.13.100.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.13.100.3@KRBTEST.COM; defaulting to no policy
I20260929 18:38:20.353781 24013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:20.342041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.2:36915 (local address 127.13.100.62:33821)
0929 18:38:20.342211 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0929 18:38:20.342216 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:38:20.343017 (+ 801us) server_negotiation.cc:407] Connection header received
0929 18:38:20.344395 (+ 1378us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:38:20.344399 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:38:20.344450 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:38:20.344578 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:38:20.345698 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:20.346292 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:20.347407 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:20.347630 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:20.347749 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:20.350119 (+ 2370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:38:20.350129 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:38:20.350131 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:38:20.350154 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:38:20.351573 (+ 1419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:38:20.352146 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:38:20.352150 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:38:20.352152 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:38:20.352236 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:38:20.352571 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:38:20.352576 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:38:20.352578 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:38:20.352824 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:38:20.352937 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 18:38:20.353424 (+ 487us) server_negotiation.cc:299] Negotiation successful
0929 18:38:20.353546 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.100.3@KRBTEST.COM" created.
I20260929 18:38:20.354600 24301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:20.342443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:33821 (local address 127.13.100.2:36915)
0929 18:38:20.342888 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:20.342937 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 18:38:20.344192 (+ 1255us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:38:20.344744 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:38:20.344750 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:38:20.345088 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:38:20.345542 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:20.345554 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:20.346415 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:20.346418 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:20.347275 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:20.347284 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:20.347781 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:20.347786 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:20.347956 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:20.348542 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:38:20.348557 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:38:20.349967 (+ 1410us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:38:20.351770 (+ 1803us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:38:20.351776 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:38:20.351791 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:38:20.352038 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:38:20.352335 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:38:20.352338 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:38:20.352340 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:38:20.352476 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:38:20.352933 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:38:20.352939 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:38:20.353194 (+ 255us) client_negotiation.cc:769] Sending connection context
0929 18:38:20.353383 (+ 189us) client_negotiation.cc:240] Negotiation successful
0929 18:38:20.353696 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":121,"threads_started":1}
I20260929 18:38:20.355923 24299 heartbeater.cc:347] Connected to a master server at 127.13.100.62:33821
I20260929 18:38:20.356190 24299 heartbeater.cc:464] Registering TS with master...
I20260929 18:38:20.356827 24299 heartbeater.cc:511] Master 127.13.100.62:33821 requested a full tablet report, sending...
I20260929 18:38:20.357928 23974 ts_manager.cc:194] Registered new tserver with Master: 077eb898bb964ae48b0cc63fe3422229 (127.13.100.2:40925)
I20260929 18:38:20.358569 23974 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.100.2@KRBTEST.COM'} at 127.13.100.2:36915
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.100.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.3.keytab.
Entry for principal kudu/127.13.100.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.3.keytab.
I20260929 18:38:20.369124 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707036041078-13712-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707036041078-13712-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707036041078-13712-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707036041078-13712-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.100.3:0
--local_ip_for_outbound_sockets=127.13.100.3
--webserver_interface=127.13.100.3
--webserver_port=0
--tserver_master_addrs=127.13.100.62:33821
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.3.keytab
--principal=kudu/127.13.100.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.100.20:42683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707036041078-13712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707036041078-13712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707036041078-13712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:38:20.477012 24304 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:38:20.477249 24304 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:38:20.477290 24304 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:38:20.481230 24304 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:38:20.481284 24304 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:38:20.481340 24304 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.100.3
I20260929 18:38:20.486545 24304 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.100.20:42683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707036041078-13712-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707036041078-13712-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.3.keytab
--principal=kudu/127.13.100.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.100.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707036041078-13712-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.100.3
--webserver_port=0
--tserver_master_addrs=127.13.100.62:33821
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24304
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.100.3
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707036041078-13712-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:38:20.487509 24304 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:38:20.488461 24304 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:38:20.497272 24312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:38:20.497303 24310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:38:20.497444 24309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:38:20.497699 24304 server_base.cc:1034] running on GCE node
I20260929 18:38:20.498106 24304 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:38:20.498718 24304 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:38:20.499867 24304 hybrid_clock.cc:648] HybridClock initialized: now 1790707100499840 us; error 39 us; skew 500 ppm
Sep 29 18:38:20 dist-test-slave-j1xs krb5kdc[23929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707100, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:38:20.502383 24304 init.cc:377] Logged in from keytab as kudu/127.13.100.3@KRBTEST.COM (short username kudu)
I20260929 18:38:20.503233 24304 webserver.cc:460] Webserver started at http://127.13.100.3:36481/ using document root <none> and password file <none>
I20260929 18:38:20.503767 24304 fs_manager.cc:359] Metadata directory not provided
I20260929 18:38:20.503854 24304 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:38:20.504066 24304 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:38:20.506062 24304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707036041078-13712-0/minicluster-data/ts-2/data/instance:
uuid: "57540a06728141f8b75e75a035bc3a15"
format_stamp: "Formatted at 2026-09-29 18:38:20 on dist-test-slave-j1xs"
server_key: "e9d2e2a0e97df1b8d7419844fe05a852"
server_key_iv: "9ea3112923919c7d6ed0cfe7b8c1e973"
server_key_version: "encryptionkey@0"
I20260929 18:38:20.506590 24304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707036041078-13712-0/minicluster-data/ts-2/wal/instance:
uuid: "57540a06728141f8b75e75a035bc3a15"
format_stamp: "Formatted at 2026-09-29 18:38:20 on dist-test-slave-j1xs"
server_key: "e9d2e2a0e97df1b8d7419844fe05a852"
server_key_iv: "9ea3112923919c7d6ed0cfe7b8c1e973"
server_key_version: "encryptionkey@0"
I20260929 18:38:20.510615 24304 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 18:38:20.512913 24319 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:38:20.514221 24304 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:38:20.514340 24304 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707036041078-13712-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707036041078-13712-0/minicluster-data/ts-2/wal
uuid: "57540a06728141f8b75e75a035bc3a15"
format_stamp: "Formatted at 2026-09-29 18:38:20 on dist-test-slave-j1xs"
server_key: "e9d2e2a0e97df1b8d7419844fe05a852"
server_key_iv: "9ea3112923919c7d6ed0cfe7b8c1e973"
server_key_version: "encryptionkey@0"
I20260929 18:38:20.514452 24304 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707036041078-13712-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707036041078-13712-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707036041078-13712-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:38:20.524345 24304 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:38:20.526913 24304 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:38:20.527099 24304 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:38:20.527612 24304 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:38:20.528349 24304 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:38:20.528421 24304 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:38:20.528484 24304 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:38:20.528532 24304 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:38:20.536844 24304 rpc_server.cc:307] RPC server started. Bound to: 127.13.100.3:44521
I20260929 18:38:20.536875 24432 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.100.3:44521 every 8 connection(s)
I20260929 18:38:20.537703 24304 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707036041078-13712-0/minicluster-data/ts-2/data/info.pb
I20260929 18:38:20.545032 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 24304
I20260929 18:38:20.545167 13712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707036041078-13712-0/minicluster-data/ts-2/wal/instance
I20260929 18:38:20.545486 13712 external_mini_cluster.cc:1428] Setting key c3f8c88ac357db92fd6bb26ed42f8278
Sep 29 18:38:20 dist-test-slave-j1xs krb5kdc[23929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707100, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.3@KRBTEST.COM for kudu/127.13.100.62@KRBTEST.COM
I20260929 18:38:20.550277 24013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:20.539334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.3:37163 (local address 127.13.100.62:33821)
0929 18:38:20.539476 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0929 18:38:20.539481 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:38:20.540267 (+ 786us) server_negotiation.cc:407] Connection header received
0929 18:38:20.541537 (+ 1270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:38:20.541540 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:38:20.541586 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:38:20.541659 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:38:20.542711 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:20.543223 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:20.544245 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:20.544456 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:20.544533 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:20.547174 (+ 2641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:38:20.547178 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:38:20.547180 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:38:20.547215 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:38:20.548498 (+ 1283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:38:20.548991 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:38:20.548994 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:38:20.548995 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:38:20.549064 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:38:20.549359 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:38:20.549361 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:38:20.549362 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:38:20.549516 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:38:20.549586 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 18:38:20.550069 (+ 483us) server_negotiation.cc:299] Negotiation successful
0929 18:38:20.550158 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 18:38:20.551074 24435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:20.539721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:33821 (local address 127.13.100.3:37163)
0929 18:38:20.540124 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:20.540162 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 18:38:20.541343 (+ 1181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:38:20.541803 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:38:20.541809 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:38:20.542140 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:38:20.542579 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:20.542591 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:20.543328 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:20.543331 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:20.544144 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:20.544151 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:20.544574 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:20.544576 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:20.544700 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:20.545381 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:38:20.545401 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:38:20.547022 (+ 1621us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:38:20.548615 (+ 1593us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:38:20.548619 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:38:20.548629 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:38:20.548897 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:38:20.549142 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:38:20.549144 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:38:20.549145 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:38:20.549278 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:38:20.549582 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:38:20.549586 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:38:20.549793 (+ 207us) client_negotiation.cc:769] Sending connection context
0929 18:38:20.550030 (+ 237us) client_negotiation.cc:240] Negotiation successful
0929 18:38:20.550281 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":97,"threads_started":1}
I20260929 18:38:20.552147 24433 heartbeater.cc:347] Connected to a master server at 127.13.100.62:33821
I20260929 18:38:20.552356 24433 heartbeater.cc:464] Registering TS with master...
I20260929 18:38:20.552768 24433 heartbeater.cc:511] Master 127.13.100.62:33821 requested a full tablet report, sending...
I20260929 18:38:20.553694 23974 ts_manager.cc:194] Registered new tserver with Master: 57540a06728141f8b75e75a035bc3a15 (127.13.100.3:44521)
I20260929 18:38:20.554348 23974 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.100.3@KRBTEST.COM'} at 127.13.100.3:37163
I20260929 18:38:20.559082 13712 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 18:38:20.567446 24013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:20.560896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42806 (local address 127.13.100.62:33821)
0929 18:38:20.561005 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0929 18:38:20.561008 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:38:20.561147 (+ 139us) server_negotiation.cc:407] Connection header received
0929 18:38:20.561227 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:38:20.561229 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:38:20.561262 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:38:20.561324 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:38:20.562030 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:20.562508 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:20.563256 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:20.563447 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:20.563519 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:20.564708 (+ 1189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:38:20.564711 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:38:20.564713 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:38:20.564732 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:38:20.566165 (+ 1433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:38:20.566517 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:38:20.566519 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:38:20.566520 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:38:20.566587 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:38:20.566797 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:38:20.566799 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:38:20.566800 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:38:20.566981 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:38:20.567078 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 18:38:20.567256 (+ 178us) server_negotiation.cc:299] Negotiation successful
0929 18:38:20.567339 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 18:38:20.570325 23974 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42806:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:38:20.573045 23974 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:38:20.586316 24444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:20.580674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.2:40925 (local address 127.0.0.1:41800)
0929 18:38:20.580975 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:20.581021 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 18:38:20.581144 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:38:20.581709 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:38:20.581715 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:38:20.581739 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:38:20.581873 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:20.581879 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:20.583472 (+ 1593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:20.583475 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:20.584749 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:20.584759 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:20.585771 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:20.585774 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:20.585876 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:20.585941 (+ 65us) client_negotiation.cc:769] Sending connection context
0929 18:38:20.586049 (+ 108us) client_negotiation.cc:240] Negotiation successful
0929 18:38:20.586162 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"mutex_wait_us":26,"thread_start_us":68,"threads_started":1}
I20260929 18:38:20.586537 24448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:20.581477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55314 (local address 127.13.100.1:38125)
0929 18:38:20.581988 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0929 18:38:20.582001 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 18:38:20.582023 (+ 22us) server_negotiation.cc:407] Connection header received
0929 18:38:20.582089 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:38:20.582097 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:38:20.582286 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:38:20.582434 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:38:20.582665 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:20.583646 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:20.585148 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:20.585964 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:20.586088 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:20.586208 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0929 18:38:20.586315 (+ 107us) server_negotiation.cc:299] Negotiation successful
0929 18:38:20.586397 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":103,"threads_started":1}
I20260929 18:38:20.586717 24447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:20.581320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.3:44521 (local address 127.0.0.1:48956)
0929 18:38:20.582128 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:20.582142 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:38:20.582221 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:38:20.582834 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:38:20.582837 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:38:20.582853 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:38:20.582935 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:20.582944 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:20.584113 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:20.584117 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:20.585242 (+ 1125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:20.585249 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:20.586302 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:20.586306 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:20.586415 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:20.586447 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 18:38:20.586512 (+ 65us) client_negotiation.cc:240] Negotiation successful
0929 18:38:20.586573 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":742,"mutex_wait_us":28,"thread_start_us":88,"threads_started":1}
I20260929 18:38:20.586483 24446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:20.581038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.1:38125 (local address 127.0.0.1:55314)
0929 18:38:20.581611 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:20.581626 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 18:38:20.581702 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:38:20.582424 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:38:20.582427 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:38:20.582448 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:38:20.582543 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:20.582549 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:20.583769 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:20.583775 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:20.585027 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:20.585037 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:20.586089 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:20.586093 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:20.586176 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:20.586198 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 18:38:20.586287 (+ 89us) client_negotiation.cc:240] Negotiation successful
0929 18:38:20.586354 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"mutex_wait_us":57,"thread_start_us":76,"threads_started":1}
I20260929 18:38:20.587113 24449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:20.581748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48956 (local address 127.13.100.3:44521)
0929 18:38:20.582426 (+ 678us) server_negotiation.cc:206] Beginning negotiation
0929 18:38:20.582437 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 18:38:20.582456 (+ 19us) server_negotiation.cc:407] Connection header received
0929 18:38:20.582525 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:38:20.582530 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:38:20.582702 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:38:20.582842 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:38:20.583050 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:20.583977 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:20.585347 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:20.586037 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:20.586125 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:20.586245 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0929 18:38:20.586875 (+ 630us) server_negotiation.cc:299] Negotiation successful
0929 18:38:20.586977 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":560,"thread_start_us":160,"threads_started":1}
I20260929 18:38:20.587452 24445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:20.580798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41800 (local address 127.13.100.2:40925)
0929 18:38:20.581191 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0929 18:38:20.581204 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 18:38:20.581232 (+ 28us) server_negotiation.cc:407] Connection header received
0929 18:38:20.581308 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:38:20.581315 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:38:20.581522 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:38:20.582051 (+ 529us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:38:20.582198 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:20.583336 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:20.584884 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:20.585637 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:20.585754 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:20.585921 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0929 18:38:20.586998 (+ 1077us) server_negotiation.cc:299] Negotiation successful
0929 18:38:20.587087 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":95,"threads_started":1}
I20260929 18:38:20.589823 24233 tablet_service.cc:1467] Processing CreateTablet for tablet 07d53656302e41e390c0ab33c327cd0b (DEFAULT_TABLE table=test-table [id=ed9bf27cbc8048f582b5de94c1fd9a02]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:38:20.589869 24099 tablet_service.cc:1467] Processing CreateTablet for tablet 07d53656302e41e390c0ab33c327cd0b (DEFAULT_TABLE table=test-table [id=ed9bf27cbc8048f582b5de94c1fd9a02]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:38:20.589982 24367 tablet_service.cc:1467] Processing CreateTablet for tablet 07d53656302e41e390c0ab33c327cd0b (DEFAULT_TABLE table=test-table [id=ed9bf27cbc8048f582b5de94c1fd9a02]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:38:20.590813 24233 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07d53656302e41e390c0ab33c327cd0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:38:20.590883 24099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07d53656302e41e390c0ab33c327cd0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:38:20.590886 24367 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07d53656302e41e390c0ab33c327cd0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:38:20.596721 24450 tablet_bootstrap.cc:492] T 07d53656302e41e390c0ab33c327cd0b P 077eb898bb964ae48b0cc63fe3422229: Bootstrap starting.
I20260929 18:38:20.598047 24451 tablet_bootstrap.cc:492] T 07d53656302e41e390c0ab33c327cd0b P 7f3ea37b5e0945599a09cc5f1e352e49: Bootstrap starting.
I20260929 18:38:20.598489 24452 tablet_bootstrap.cc:492] T 07d53656302e41e390c0ab33c327cd0b P 57540a06728141f8b75e75a035bc3a15: Bootstrap starting.
I20260929 18:38:20.599344 24450 tablet_bootstrap.cc:654] T 07d53656302e41e390c0ab33c327cd0b P 077eb898bb964ae48b0cc63fe3422229: Neither blocks nor log segments found. Creating new log.
I20260929 18:38:20.600216 24450 log.cc:824] T 07d53656302e41e390c0ab33c327cd0b P 077eb898bb964ae48b0cc63fe3422229: Log is configured to *not* fsync() on all Append() calls
I20260929 18:38:20.600462 24452 tablet_bootstrap.cc:654] T 07d53656302e41e390c0ab33c327cd0b P 57540a06728141f8b75e75a035bc3a15: Neither blocks nor log segments found. Creating new log.
I20260929 18:38:20.600507 24451 tablet_bootstrap.cc:654] T 07d53656302e41e390c0ab33c327cd0b P 7f3ea37b5e0945599a09cc5f1e352e49: Neither blocks nor log segments found. Creating new log.
I20260929 18:38:20.601390 24451 log.cc:824] T 07d53656302e41e390c0ab33c327cd0b P 7f3ea37b5e0945599a09cc5f1e352e49: Log is configured to *not* fsync() on all Append() calls
I20260929 18:38:20.601627 24452 log.cc:824] T 07d53656302e41e390c0ab33c327cd0b P 57540a06728141f8b75e75a035bc3a15: Log is configured to *not* fsync() on all Append() calls
I20260929 18:38:20.603070 24450 tablet_bootstrap.cc:492] T 07d53656302e41e390c0ab33c327cd0b P 077eb898bb964ae48b0cc63fe3422229: No bootstrap required, opened a new log
I20260929 18:38:20.603377 24450 ts_tablet_manager.cc:1397] T 07d53656302e41e390c0ab33c327cd0b P 077eb898bb964ae48b0cc63fe3422229: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260929 18:38:20.603570 24452 tablet_bootstrap.cc:492] T 07d53656302e41e390c0ab33c327cd0b P 57540a06728141f8b75e75a035bc3a15: No bootstrap required, opened a new log
I20260929 18:38:20.603760 24452 ts_tablet_manager.cc:1397] T 07d53656302e41e390c0ab33c327cd0b P 57540a06728141f8b75e75a035bc3a15: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260929 18:38:20.603857 24451 tablet_bootstrap.cc:492] T 07d53656302e41e390c0ab33c327cd0b P 7f3ea37b5e0945599a09cc5f1e352e49: No bootstrap required, opened a new log
I20260929 18:38:20.604028 24451 ts_tablet_manager.cc:1397] T 07d53656302e41e390c0ab33c327cd0b P 7f3ea37b5e0945599a09cc5f1e352e49: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 18:38:20.606349 24450 raft_consensus.cc:348] T 07d53656302e41e390c0ab33c327cd0b P 077eb898bb964ae48b0cc63fe3422229 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "077eb898bb964ae48b0cc63fe3422229" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 40925 } } peers { permanent_uuid: "57540a06728141f8b75e75a035bc3a15" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 44521 } } peers { permanent_uuid: "7f3ea37b5e0945599a09cc5f1e352e49" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 38125 } }
I20260929 18:38:20.606573 24450 raft_consensus.cc:374] T 07d53656302e41e390c0ab33c327cd0b P 077eb898bb964ae48b0cc63fe3422229 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:38:20.606647 24450 raft_consensus.cc:729] T 07d53656302e41e390c0ab33c327cd0b P 077eb898bb964ae48b0cc63fe3422229 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 077eb898bb964ae48b0cc63fe3422229, State: Initialized, Role: FOLLOWER
I20260929 18:38:20.607086 24450 consensus_queue.cc:260] T 07d53656302e41e390c0ab33c327cd0b P 077eb898bb964ae48b0cc63fe3422229 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "077eb898bb964ae48b0cc63fe3422229" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 40925 } } peers { permanent_uuid: "57540a06728141f8b75e75a035bc3a15" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 44521 } } peers { permanent_uuid: "7f3ea37b5e0945599a09cc5f1e352e49" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 38125 } }
I20260929 18:38:20.607481 24452 raft_consensus.cc:348] T 07d53656302e41e390c0ab33c327cd0b P 57540a06728141f8b75e75a035bc3a15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "077eb898bb964ae48b0cc63fe3422229" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 40925 } } peers { permanent_uuid: "57540a06728141f8b75e75a035bc3a15" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 44521 } } peers { permanent_uuid: "7f3ea37b5e0945599a09cc5f1e352e49" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 38125 } }
I20260929 18:38:20.607510 24451 raft_consensus.cc:348] T 07d53656302e41e390c0ab33c327cd0b P 7f3ea37b5e0945599a09cc5f1e352e49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "077eb898bb964ae48b0cc63fe3422229" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 40925 } } peers { permanent_uuid: "57540a06728141f8b75e75a035bc3a15" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 44521 } } peers { permanent_uuid: "7f3ea37b5e0945599a09cc5f1e352e49" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 38125 } }
I20260929 18:38:20.607688 24299 heartbeater.cc:503] Master 127.13.100.62:33821 was elected leader, sending a full tablet report...
I20260929 18:38:20.607736 24451 raft_consensus.cc:374] T 07d53656302e41e390c0ab33c327cd0b P 7f3ea37b5e0945599a09cc5f1e352e49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:38:20.607820 24451 raft_consensus.cc:729] T 07d53656302e41e390c0ab33c327cd0b P 7f3ea37b5e0945599a09cc5f1e352e49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f3ea37b5e0945599a09cc5f1e352e49, State: Initialized, Role: FOLLOWER
I20260929 18:38:20.607827 24452 raft_consensus.cc:374] T 07d53656302e41e390c0ab33c327cd0b P 57540a06728141f8b75e75a035bc3a15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:38:20.607877 24452 raft_consensus.cc:729] T 07d53656302e41e390c0ab33c327cd0b P 57540a06728141f8b75e75a035bc3a15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57540a06728141f8b75e75a035bc3a15, State: Initialized, Role: FOLLOWER
I20260929 18:38:20.608029 24450 ts_tablet_manager.cc:1428] T 07d53656302e41e390c0ab33c327cd0b P 077eb898bb964ae48b0cc63fe3422229: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260929 18:38:20.608291 24451 consensus_queue.cc:260] T 07d53656302e41e390c0ab33c327cd0b P 7f3ea37b5e0945599a09cc5f1e352e49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "077eb898bb964ae48b0cc63fe3422229" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 40925 } } peers { permanent_uuid: "57540a06728141f8b75e75a035bc3a15" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 44521 } } peers { permanent_uuid: "7f3ea37b5e0945599a09cc5f1e352e49" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 38125 } }
I20260929 18:38:20.608336 24452 consensus_queue.cc:260] T 07d53656302e41e390c0ab33c327cd0b P 57540a06728141f8b75e75a035bc3a15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "077eb898bb964ae48b0cc63fe3422229" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 40925 } } peers { permanent_uuid: "57540a06728141f8b75e75a035bc3a15" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 44521 } } peers { permanent_uuid: "7f3ea37b5e0945599a09cc5f1e352e49" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 38125 } }
I20260929 18:38:20.608925 24165 heartbeater.cc:503] Master 127.13.100.62:33821 was elected leader, sending a full tablet report...
I20260929 18:38:20.609162 24433 heartbeater.cc:503] Master 127.13.100.62:33821 was elected leader, sending a full tablet report...
I20260929 18:38:20.609447 24451 ts_tablet_manager.cc:1428] T 07d53656302e41e390c0ab33c327cd0b P 7f3ea37b5e0945599a09cc5f1e352e49: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260929 18:38:20.610205 24452 ts_tablet_manager.cc:1428] T 07d53656302e41e390c0ab33c327cd0b P 57540a06728141f8b75e75a035bc3a15: Time spent starting tablet: real 0.006s user 0.002s sys 0.005s
W20260929 18:38:20.631342 24166 tablet.cc:2369] T 07d53656302e41e390c0ab33c327cd0b P 7f3ea37b5e0945599a09cc5f1e352e49: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 18:38:20.788647 24434 tablet.cc:2369] T 07d53656302e41e390c0ab33c327cd0b P 57540a06728141f8b75e75a035bc3a15: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:38:20.838140 24457 raft_consensus.cc:482] T 07d53656302e41e390c0ab33c327cd0b P 7f3ea37b5e0945599a09cc5f1e352e49 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:38:20.838336 24457 raft_consensus.cc:504] T 07d53656302e41e390c0ab33c327cd0b P 7f3ea37b5e0945599a09cc5f1e352e49 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "077eb898bb964ae48b0cc63fe3422229" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 40925 } } peers { permanent_uuid: "57540a06728141f8b75e75a035bc3a15" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 44521 } } peers { permanent_uuid: "7f3ea37b5e0945599a09cc5f1e352e49" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 38125 } }
I20260929 18:38:20.839387 24457 leader_election.cc:288] T 07d53656302e41e390c0ab33c327cd0b P 7f3ea37b5e0945599a09cc5f1e352e49 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 077eb898bb964ae48b0cc63fe3422229 (127.13.100.2:40925), 57540a06728141f8b75e75a035bc3a15 (127.13.100.3:44521)
W20260929 18:38:20.841254 24300 tablet.cc:2369] T 07d53656302e41e390c0ab33c327cd0b P 077eb898bb964ae48b0cc63fe3422229: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:38:20.843784 24445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:20.839754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.1:43213 (local address 127.13.100.2:40925)
0929 18:38:20.839919 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0929 18:38:20.839924 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:38:20.840041 (+ 117us) server_negotiation.cc:407] Connection header received
0929 18:38:20.840195 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:38:20.840201 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:38:20.840254 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:38:20.840331 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:38:20.840641 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:20.841430 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:20.842611 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:20.843222 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:20.843361 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:20.843411 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0929 18:38:20.843601 (+ 190us) server_negotiation.cc:299] Negotiation successful
0929 18:38:20.843667 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 18:38:20.843807 24459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:20.839648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.2:40925 (local address 127.13.100.1:43213)
0929 18:38:20.839933 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:20.839950 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 18:38:20.840093 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:38:20.840367 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:38:20.840370 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:38:20.840394 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:38:20.840529 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:20.840537 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:20.841542 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:20.841545 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:20.842486 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:20.842495 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:20.843362 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:20.843365 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:20.843468 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:20.843490 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 18:38:20.843586 (+ 96us) client_negotiation.cc:240] Negotiation successful
0929 18:38:20.843655 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"mutex_wait_us":41,"thread_start_us":100,"threads_started":1}
I20260929 18:38:20.843907 24449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:20.840164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.1:58923 (local address 127.13.100.3:44521)
0929 18:38:20.840294 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0929 18:38:20.840300 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:38:20.840483 (+ 183us) server_negotiation.cc:407] Connection header received
0929 18:38:20.840599 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:38:20.840604 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:38:20.840654 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:38:20.840730 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:38:20.840956 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:20.841644 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:20.842687 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:20.843252 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:20.843362 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:20.843411 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0929 18:38:20.843720 (+ 309us) server_negotiation.cc:299] Negotiation successful
0929 18:38:20.843788 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 18:38:20.844235 24460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:20.840052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.3:44521 (local address 127.13.100.1:58923)
0929 18:38:20.840382 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:20.840394 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:38:20.840481 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:38:20.840748 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:38:20.840750 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:38:20.840766 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:38:20.840853 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:20.840857 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:20.841754 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:20.841758 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:20.842563 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:38:20.842573 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:38:20.843470 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:38:20.843474 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:38:20.843589 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:20.843609 (+ 20us) client_negotiation.cc:769] Sending connection context
0929 18:38:20.844026 (+ 417us) client_negotiation.cc:240] Negotiation successful
0929 18:38:20.844104 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"mutex_wait_us":16,"thread_start_us":91,"threads_started":1}
I20260929 18:38:20.844520 24253 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07d53656302e41e390c0ab33c327cd0b" candidate_uuid: "7f3ea37b5e0945599a09cc5f1e352e49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "077eb898bb964ae48b0cc63fe3422229" is_pre_election: true
I20260929 18:38:20.844847 24253 raft_consensus.cc:2454] T 07d53656302e41e390c0ab33c327cd0b P 077eb898bb964ae48b0cc63fe3422229 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f3ea37b5e0945599a09cc5f1e352e49 in term 0.
I20260929 18:38:20.845216 24387 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07d53656302e41e390c0ab33c327cd0b" candidate_uuid: "7f3ea37b5e0945599a09cc5f1e352e49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57540a06728141f8b75e75a035bc3a15" is_pre_election: true
I20260929 18:38:20.845348 24053 leader_election.cc:302] T 07d53656302e41e390c0ab33c327cd0b P 7f3ea37b5e0945599a09cc5f1e352e49 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 077eb898bb964ae48b0cc63fe3422229, 7f3ea37b5e0945599a09cc5f1e352e49; no voters:
I20260929 18:38:20.845504 24387 raft_consensus.cc:2454] T 07d53656302e41e390c0ab33c327cd0b P 57540a06728141f8b75e75a035bc3a15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f3ea37b5e0945599a09cc5f1e352e49 in term 0.
I20260929 18:38:20.845628 24457 raft_consensus.cc:2781] T 07d53656302e41e390c0ab33c327cd0b P 7f3ea37b5e0945599a09cc5f1e352e49 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:38:20.845714 24457 raft_consensus.cc:482] T 07d53656302e41e390c0ab33c327cd0b P 7f3ea37b5e0945599a09cc5f1e352e49 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:38:20.845752 24457 raft_consensus.cc:3037] T 07d53656302e41e390c0ab33c327cd0b P 7f3ea37b5e0945599a09cc5f1e352e49 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:38:20.846984 24457 raft_consensus.cc:504] T 07d53656302e41e390c0ab33c327cd0b P 7f3ea37b5e0945599a09cc5f1e352e49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "077eb898bb964ae48b0cc63fe3422229" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 40925 } } peers { permanent_uuid: "57540a06728141f8b75e75a035bc3a15" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 44521 } } peers { permanent_uuid: "7f3ea37b5e0945599a09cc5f1e352e49" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 38125 } }
I20260929 18:38:20.847309 24457 leader_election.cc:288] T 07d53656302e41e390c0ab33c327cd0b P 7f3ea37b5e0945599a09cc5f1e352e49 [CANDIDATE]: Term 1 election: Requested vote from peers 077eb898bb964ae48b0cc63fe3422229 (127.13.100.2:40925), 57540a06728141f8b75e75a035bc3a15 (127.13.100.3:44521)
I20260929 18:38:20.847625 24253 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07d53656302e41e390c0ab33c327cd0b" candidate_uuid: "7f3ea37b5e0945599a09cc5f1e352e49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "077eb898bb964ae48b0cc63fe3422229"
I20260929 18:38:20.847756 24387 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07d53656302e41e390c0ab33c327cd0b" candidate_uuid: "7f3ea37b5e0945599a09cc5f1e352e49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57540a06728141f8b75e75a035bc3a15"
I20260929 18:38:20.847766 24253 raft_consensus.cc:3037] T 07d53656302e41e390c0ab33c327cd0b P 077eb898bb964ae48b0cc63fe3422229 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:38:20.847875 24387 raft_consensus.cc:3037] T 07d53656302e41e390c0ab33c327cd0b P 57540a06728141f8b75e75a035bc3a15 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:38:20.849052 24387 raft_consensus.cc:2454] T 07d53656302e41e390c0ab33c327cd0b P 57540a06728141f8b75e75a035bc3a15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f3ea37b5e0945599a09cc5f1e352e49 in term 1.
I20260929 18:38:20.849052 24253 raft_consensus.cc:2454] T 07d53656302e41e390c0ab33c327cd0b P 077eb898bb964ae48b0cc63fe3422229 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f3ea37b5e0945599a09cc5f1e352e49 in term 1.
I20260929 18:38:20.849313 24053 leader_election.cc:302] T 07d53656302e41e390c0ab33c327cd0b P 7f3ea37b5e0945599a09cc5f1e352e49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57540a06728141f8b75e75a035bc3a15, 7f3ea37b5e0945599a09cc5f1e352e49; no voters:
I20260929 18:38:20.849511 24457 raft_consensus.cc:2781] T 07d53656302e41e390c0ab33c327cd0b P 7f3ea37b5e0945599a09cc5f1e352e49 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:38:20.849748 24457 raft_consensus.cc:686] T 07d53656302e41e390c0ab33c327cd0b P 7f3ea37b5e0945599a09cc5f1e352e49 [term 1 LEADER]: Becoming Leader. State: Replica: 7f3ea37b5e0945599a09cc5f1e352e49, State: Running, Role: LEADER
I20260929 18:38:20.850085 24457 consensus_queue.cc:237] T 07d53656302e41e390c0ab33c327cd0b P 7f3ea37b5e0945599a09cc5f1e352e49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "077eb898bb964ae48b0cc63fe3422229" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 40925 } } peers { permanent_uuid: "57540a06728141f8b75e75a035bc3a15" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 44521 } } peers { permanent_uuid: "7f3ea37b5e0945599a09cc5f1e352e49" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 38125 } }
I20260929 18:38:20.853267 23972 catalog_manager.cc:5495] T 07d53656302e41e390c0ab33c327cd0b P 7f3ea37b5e0945599a09cc5f1e352e49 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7f3ea37b5e0945599a09cc5f1e352e49 (127.13.100.1). New cstate: current_term: 1 leader_uuid: "7f3ea37b5e0945599a09cc5f1e352e49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "077eb898bb964ae48b0cc63fe3422229" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 40925 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "57540a06728141f8b75e75a035bc3a15" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 44521 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7f3ea37b5e0945599a09cc5f1e352e49" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 38125 } health_report { overall_health: HEALTHY } } }
I20260929 18:38:20.878155 24448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:20.875145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55316 (local address 127.13.100.1:38125)
0929 18:38:20.875266 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0929 18:38:20.875270 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:38:20.875283 (+ 13us) server_negotiation.cc:407] Connection header received
0929 18:38:20.875369 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:38:20.875371 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:38:20.875421 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:38:20.875512 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:38:20.875705 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:20.876181 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:20.876942 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:38:20.877134 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:38:20.877205 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:38:20.877436 (+ 231us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:38:20.877810 (+ 374us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:38:20.877950 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0929 18:38:20.878009 (+ 59us) server_negotiation.cc:299] Negotiation successful
0929 18:38:20.878055 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 18:38:20.883582 24253 raft_consensus.cc:1261] T 07d53656302e41e390c0ab33c327cd0b P 077eb898bb964ae48b0cc63fe3422229 [term 1 FOLLOWER]: Refusing update from remote peer 7f3ea37b5e0945599a09cc5f1e352e49: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:38:20.883579 24387 raft_consensus.cc:1261] T 07d53656302e41e390c0ab33c327cd0b P 57540a06728141f8b75e75a035bc3a15 [term 1 FOLLOWER]: Refusing update from remote peer 7f3ea37b5e0945599a09cc5f1e352e49: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:38:20.884267 24461 consensus_queue.cc:1035] T 07d53656302e41e390c0ab33c327cd0b P 7f3ea37b5e0945599a09cc5f1e352e49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "57540a06728141f8b75e75a035bc3a15" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 44521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:38:20.884405 24457 consensus_queue.cc:1035] T 07d53656302e41e390c0ab33c327cd0b P 7f3ea37b5e0945599a09cc5f1e352e49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "077eb898bb964ae48b0cc63fe3422229" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 40925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:38:20.900833 23972 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42806:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:38:20.901007 23972 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42806:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:38:20.903303 23972 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c7182844bcef446998295760fc1bb3cd: Sending DeleteTablet for 3 replicas of tablet 07d53656302e41e390c0ab33c327cd0b
I20260929 18:38:20.904008 24233 tablet_service.cc:1514] Processing DeleteTablet for tablet 07d53656302e41e390c0ab33c327cd0b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:38:20 UTC) from {username='kudu', principal='kudu/127.13.100.62@KRBTEST.COM'} at 127.0.0.1:41800
I20260929 18:38:20.904250 24099 tablet_service.cc:1514] Processing DeleteTablet for tablet 07d53656302e41e390c0ab33c327cd0b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:38:20 UTC) from {username='kudu', principal='kudu/127.13.100.62@KRBTEST.COM'} at 127.0.0.1:55314
I20260929 18:38:20.904528 24367 tablet_service.cc:1514] Processing DeleteTablet for tablet 07d53656302e41e390c0ab33c327cd0b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:38:20 UTC) from {username='kudu', principal='kudu/127.13.100.62@KRBTEST.COM'} at 127.0.0.1:48956
I20260929 18:38:20.904603 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 24036
I20260929 18:38:20.905308 24473 tablet_replica.cc:330] T 07d53656302e41e390c0ab33c327cd0b P 077eb898bb964ae48b0cc63fe3422229: stopping tablet replica
I20260929 18:38:20.905642 24473 raft_consensus.cc:2229] T 07d53656302e41e390c0ab33c327cd0b P 077eb898bb964ae48b0cc63fe3422229 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:38:20.905995 24473 raft_consensus.cc:2258] T 07d53656302e41e390c0ab33c327cd0b P 077eb898bb964ae48b0cc63fe3422229 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:38:20.907188 24475 tablet_replica.cc:330] T 07d53656302e41e390c0ab33c327cd0b P 57540a06728141f8b75e75a035bc3a15: stopping tablet replica
I20260929 18:38:20.907466 24475 raft_consensus.cc:2229] T 07d53656302e41e390c0ab33c327cd0b P 57540a06728141f8b75e75a035bc3a15 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:38:20.907482 24473 ts_tablet_manager.cc:1905] T 07d53656302e41e390c0ab33c327cd0b P 077eb898bb964ae48b0cc63fe3422229: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:38:20.907761 24475 raft_consensus.cc:2258] T 07d53656302e41e390c0ab33c327cd0b P 57540a06728141f8b75e75a035bc3a15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:38:20.910874 24475 ts_tablet_manager.cc:1905] T 07d53656302e41e390c0ab33c327cd0b P 57540a06728141f8b75e75a035bc3a15: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:38:20.911100 24473 ts_tablet_manager.cc:1918] T 07d53656302e41e390c0ab33c327cd0b P 077eb898bb964ae48b0cc63fe3422229: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:38:20.911243 24473 log.cc:1196] T 07d53656302e41e390c0ab33c327cd0b P 077eb898bb964ae48b0cc63fe3422229: Deleting WAL directory at /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707036041078-13712-0/minicluster-data/ts-1/wal/wals/07d53656302e41e390c0ab33c327cd0b
I20260929 18:38:20.911603 24473 ts_tablet_manager.cc:1939] T 07d53656302e41e390c0ab33c327cd0b P 077eb898bb964ae48b0cc63fe3422229: Deleting consensus metadata
I20260929 18:38:20.912920 23959 catalog_manager.cc:4841] TS 077eb898bb964ae48b0cc63fe3422229 (127.13.100.2:40925): tablet 07d53656302e41e390c0ab33c327cd0b (table test-table [id=ed9bf27cbc8048f582b5de94c1fd9a02]) successfully deleted
I20260929 18:38:20.913164 24475 ts_tablet_manager.cc:1918] T 07d53656302e41e390c0ab33c327cd0b P 57540a06728141f8b75e75a035bc3a15: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:38:20.913293 24475 log.cc:1196] T 07d53656302e41e390c0ab33c327cd0b P 57540a06728141f8b75e75a035bc3a15: Deleting WAL directory at /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707036041078-13712-0/minicluster-data/ts-2/wal/wals/07d53656302e41e390c0ab33c327cd0b
W20260929 18:38:20.913417 23961 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.100.1:38125 (error 108)
I20260929 18:38:20.913621 24475 ts_tablet_manager.cc:1939] T 07d53656302e41e390c0ab33c327cd0b P 57540a06728141f8b75e75a035bc3a15: Deleting consensus metadata
I20260929 18:38:20.914328 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 24170
I20260929 18:38:20.914498 24444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:38:20.914228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.1:38125 (local address 127.0.0.1:55328)
0929 18:38:20.914336 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0929 18:38:20.914406 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.1:38125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260929 18:38:20.915375 23961 catalog_manager.cc:4584] TS 7f3ea37b5e0945599a09cc5f1e352e49 (127.13.100.1:38125): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 07d53656302e41e390c0ab33c327cd0b: Network error: Client connection negotiation failed: client connection to 127.13.100.1:38125: connect: Connection refused (error 111)
I20260929 18:38:20.915406 23959 catalog_manager.cc:4841] TS 57540a06728141f8b75e75a035bc3a15 (127.13.100.3:44521): tablet 07d53656302e41e390c0ab33c327cd0b (table test-table [id=ed9bf27cbc8048f582b5de94c1fd9a02]) successfully deleted
I20260929 18:38:20.920404 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 24304
I20260929 18:38:20.925735 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 23942
2026-09-29T18:38:20Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2436 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707036041078-13712-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:38:20 dist-test-slave-j1xs krb5kdc[24479](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:38:20 dist-test-slave-j1xs krb5kdc[24479](info): set up 2 sockets
Sep 29 18:38:20 dist-test-slave-j1xs krb5kdc[24479](info): commencing operation
krb5kdc: starting...
W20260929 18:38:21.970860 13712 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707036041078-13712-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707036041078-13712-0/krb5kdc/test-user.keytab.
Sep 29 18:38:22 dist-test-slave-j1xs krb5kdc[24479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707102, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:38:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:38:22Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.13.100.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.13.100.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.13.100.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707036041078-13712-0/krb5kdc/rangeradmin_127.13.100.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.13.100.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707036041078-13712-0/krb5kdc/rangeradmin_127.13.100.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.13.100.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.13.100.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.13.100.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707036041078-13712-0/krb5kdc/rangerlookup_127.13.100.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.13.100.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707036041078-13712-0/krb5kdc/rangerlookup_127.13.100.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.13.100.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.100.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.100.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707036041078-13712-0/krb5kdc/HTTP_127.13.100.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.13.100.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707036041078-13712-0/krb5kdc/HTTP_127.13.100.20@KRBTEST.COM.keytab.
I20260929 18:38:22.133121 13712 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707036041078-13712-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskpp0tHW/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707036041078-13712-0/minicluster-data/postgres -l logfile start
2026-09-29 18:38:24.590 UTC [24509] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-29 18:38:24.590 UTC [24509] LOG: listening on IPv4 address "127.13.100.20", port 57331
2026-09-29 18:38:24.594 UTC [24509] LOG: listening on Unix socket "/tmp/.s.PGSQL.57331"
2026-09-29 18:38:24.615 UTC [24512] LOG: database system was shut down at 2026-09-29 18:38:23 UTC
2026-09-29 18:38:24.619 UTC [24509] LOG: database system is ready to accept connections
I20260929 18:38:25.574572 13712 mini_postgres.cc:96] Postgres bound to 57331
2026-09-29 18:38:25.580 UTC [24520] FATAL: database "slave" does not exist
127.13.100.20:57331 - accepting connections
I20260929 18:38:25.581127 13712 mini_ranger.cc:161] Starting Ranger...
I20260929 18:38:25.595618 13712 mini_ranger.cc:84] Created miniranger Postgres user
I20260929 18:38:26.057626 13712 mini_ranger.cc:87] Created ranger Postgres database
I20260929 18:38:26.057722 13712 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707036041078-13712-0/minicluster-data/ranger-admin
2026-09-29 18:38:26,334 [I] DB FLAVOR :POSTGRES
2026-09-29 18:38:26,335 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 18:38:26,335 [I] Checking connection..
2026-09-29 18:38:26,335 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpp0tHW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskpp0tHW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.100.20:57331/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-29 18:38:26,572 [I] Checking connection passed.
2026-09-29 18:38:26,572 [I] --------- Verifying version history table ---------
2026-09-29 18:38:26,572 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpp0tHW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskpp0tHW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.100.20:57331/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 18:38:26,821 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 18:38:26,822 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpp0tHW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskpp0tHW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.100.20:57331/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 18:38:27,051 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 18:38:27,051 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-29 18:38:27,051 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpp0tHW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskpp0tHW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.100.20:57331/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskpp0tHW/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-29 18:38:27.274 UTC [24584] WARNING: there is no transaction in progress
2026-09-29 18:38:27,280 [I] create_dbversion_catalog.sql file imported successfully
2026-09-29 18:38:27,281 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpp0tHW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskpp0tHW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.100.20:57331/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 18:38:27,536 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-29 18:38:27,537 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 18:38:27,537 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpp0tHW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskpp0tHW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.100.20:57331/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-29 18:38:27,782 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpp0tHW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskpp0tHW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.100.20:57331/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-29 18:38:28,006 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpp0tHW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskpp0tHW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.100.20:57331/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-j1xs.c.gcp-upstream.internal','N') ;"
2026-09-29 18:38:28,263 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpp0tHW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskpp0tHW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.100.20:57331/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 18:38:28,490 [I] Table x_portal_user does not exist in database ranger
2026-09-29 18:38:28,490 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpp0tHW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskpp0tHW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.100.20:57331/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 18:38:28,743 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-29 18:38:28,743 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpp0tHW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskpp0tHW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.100.20:57331/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 18:38:29,003 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpp0tHW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskpp0tHW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.100.20:57331/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 18:38:29,250 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-29 18:38:29,250 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpp0tHW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskpp0tHW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.100.20:57331/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskpp0tHW/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-29 18:38:30.228 UTC [24692] WARNING: there is no transaction in progress
2026-09-29 18:38:30.244 UTC [24692] WARNING: there is no transaction in progress
2026-09-29 18:38:30.264 UTC [24692] WARNING: there is no transaction in progress
2026-09-29 18:38:30.279 UTC [24692] WARNING: there is no transaction in progress
2026-09-29 18:38:30.298 UTC [24692] WARNING: there is no transaction in progress
2026-09-29 18:38:30.317 UTC [24692] WARNING: there is no transaction in progress
2026-09-29 18:38:30.380 UTC [24692] WARNING: there is no transaction in progress
2026-09-29 18:38:30.387 UTC [24692] WARNING: there is no transaction in progress
2026-09-29 18:38:30.401 UTC [24692] WARNING: there is no transaction in progress
2026-09-29 18:38:30.410 UTC [24692] WARNING: there is no transaction in progress
2026-09-29 18:38:30.420 UTC [24692] WARNING: there is no transaction in progress
2026-09-29 18:38:30.434 UTC [24692] WARNING: there is no transaction in progress
2026-09-29 18:38:30.444 UTC [24692] WARNING: there is no transaction in progress
2026-09-29 18:38:30.450 UTC [24692] WARNING: there is no transaction in progress
2026-09-29 18:38:30.458 UTC [24692] WARNING: there is no transaction in progress
2026-09-29 18:38:30.465 UTC [24692] WARNING: there is no transaction in progress
2026-09-29 18:38:31,108 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-29 18:38:31,108 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpp0tHW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskpp0tHW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.100.20:57331/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-29 18:38:31,351 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-29 18:38:31,352 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpp0tHW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskpp0tHW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.100.20:57331/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 18:38:31,580 [I] Table x_portal_user already exists in database 'ranger'
2026-09-29 18:38:31,581 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpp0tHW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskpp0tHW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.100.20:57331/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 18:38:31,809 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-29 18:38:31,809 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpp0tHW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskpp0tHW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.100.20:57331/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 18:38:32,057 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpp0tHW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskpp0tHW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.100.20:57331/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 18:38:32,295 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpp0tHW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskpp0tHW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.100.20:57331/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-j1xs.c.gcp-upstream.internal';"
2026-09-29 18:38:32,510 [I] CORE_DB_SCHEMA import status has been updated
2026-09-29 18:38:32,510 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpp0tHW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskpp0tHW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.100.20:57331/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-29 18:38:32,756 [I] DB_PATCHES have already been applied
I20260929 18:38:32.761940 13712 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707036041078-13712-0/minicluster-data/ranger-admin:/tmp/dist-test-taskpp0tHW/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskpp0tHW/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskpp0tHW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskpp0tHW/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260929 18:38:32.762010 13712 mini_ranger.cc:193] Using host: 127.13.100.20
I20260929 18:38:32.765282 13712 mini_ranger.cc:227] Ranger admin URL: http://127.13.100.20:56875
Sep 29, 2026 6:38:33 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 6:38:33 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 6:38:33 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707036041078-13712-0/minicluster-data/ranger-admin/ews/webapp
Sep 29, 2026 6:38:33 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707036041078-13712-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 29, 2026 6:38:33 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707036041078-13712-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 29, 2026 6:38:33 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707036041078-13712-0/minicluster-data/ranger-admin/ews/webapp].
Sep 29, 2026 6:38:33 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260929 18:38:33.879284 13712 mini_ranger.cc:160] Time spent starting Ranger: real 8.298s user 0.000s sys 0.010s
Sep 29, 2026 6:38:54 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 29, 2026 6:38:54 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 29, 2026 6:38:54 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 29, 2026 6:38:54 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 29, 2026 6:38:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 29, 2026 6:38:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 29, 2026 6:38:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 29, 2026 6:38:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 29, 2026 6:38:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 29, 2026 6:38:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 29, 2026 6:38:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 29, 2026 6:38:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 29, 2026 6:38:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 29, 2026 6:38:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 29, 2026 6:38:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 29, 2026 6:38:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 29, 2026 6:38:54 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 29, 2026 6:38:54 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260929 18:38:55.447669 13712 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.13.100.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.13.100.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.13.100.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707036041078-13712-0/krb5kdc/rangerkms_127.13.100.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.13.100.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707036041078-13712-0/krb5kdc/rangerkms_127.13.100.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.13.100.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.13.100.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.100.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707036041078-13712-0/krb5kdc/HTTP_127.13.100.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.13.100.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707036041078-13712-0/krb5kdc/HTTP_127.13.100.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 29 18:38:55 dist-test-slave-j1xs krb5kdc[24479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707135, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260929 18:38:55.536808 13712 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260929 18:38:55.546837 13712 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260929 18:38:56.444715 13712 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260929 18:38:56.444841 13712 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707036041078-13712-0/minicluster-data/ranger-kms
I20260929 18:38:56.444988 13712 mini_ranger_kms.cc:227] Using postgres at 127.13.100.20:57331
2026-09-29 18:38:56,638 [I] DB FLAVOR :POSTGRES
2026-09-29 18:38:56,638 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 18:38:56,638 [I] Checking connection
2026-09-29 18:38:56,638 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpp0tHW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707036041078-13712-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.100.20:57331/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-29 18:38:56,880 [I] connection success
2026-09-29 18:38:56,880 [I] --------- Verifying Ranger DB tables ---------
2026-09-29 18:38:56,880 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-29 18:38:56,880 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpp0tHW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707036041078-13712-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.100.20:57331/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-29 18:38:57,114 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-29 18:38:57,114 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 18:38:57,114 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-29 18:38:57,114 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpp0tHW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707036041078-13712-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.100.20:57331/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707036041078-13712-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-29 18:38:57,382 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260929 18:38:57.811506 13712 mini_ranger_kms.cc:310] Created kms service
I20260929 18:38:57.984454 13712 mini_ranger_kms.cc:326] Created kudu user
I20260929 18:38:58.030352 13712 mini_ranger_kms.cc:343] Created rangerkms user
Sep 29 18:38:58 dist-test-slave-j1xs krb5kdc[24479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707135, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.13.100.20@KRBTEST.COM
I20260929 18:38:58.425544 13712 mini_ranger_kms.cc:384] Added ranger policy
I20260929 18:38:58.425704 13712 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707036041078-13712-0/minicluster-data/ranger-kms:/tmp/dist-test-taskpp0tHW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskpp0tHW/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskpp0tHW/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260929 18:38:58.425727 13712 mini_ranger_kms.cc:240] Using host: 127.13.100.20
I20260929 18:38:58.429457 13712 mini_ranger_kms.cc:276] Ranger KMS PID: 24872
I20260929 18:38:58.429602 13712 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.13.100.20:33897
Sep 29, 2026 6:38:59 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 6:38:59 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 6:38:59 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707036041078-13712-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 29, 2026 6:38:59 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707036041078-13712-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 29, 2026 6:38:59 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707036041078-13712-0/minicluster-data/ranger-kms/ews/webapp].
Sep 29, 2026 6:38:59 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260929 18:39:00.676903 13712 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.140s user 0.007s sys 0.171s
I20260929 18:39:00.677052 13712 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260929 18:39:00.677192 13712 mini_ranger_kms.cc:401] 127.13.100.20:33897/kms/v1/keys
Sep 29 18:39:00 dist-test-slave-j1xs krb5kdc[24479](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790707140, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.13.100.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 29 18:39:01 dist-test-slave-j1xs krb5kdc[24479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707140, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.13.100.20@KRBTEST.COM for HTTP/127.13.100.20@KRBTEST.COM
Sep 29 18:39:01 dist-test-slave-j1xs krb5kdc[24479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707140, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.13.100.20@KRBTEST.COM for HTTP/127.13.100.20@KRBTEST.COM
Sep 29 18:39:03 dist-test-slave-j1xs krb5kdc[24479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707143, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.13.100.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.100.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.100.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.62.keytab.
Entry for principal kudu/127.13.100.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.62.keytab.
I20260929 18:39:03.677016 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707036041078-13712-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707036041078-13712-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707036041078-13712-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.100.62:35459
--webserver_interface=127.13.100.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.62.keytab
--principal=kudu/127.13.100.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.100.20:45991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.100.62:35459
--ranger_config_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707036041078-13712-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.13.100.20:33897/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707036041078-13712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707036041078-13712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707036041078-13712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:39:03.790586 24923 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:39:03.790885 24923 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:39:03.790927 24923 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:39:03.795171 24923 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:39:03.795243 24923 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:39:03.795262 24923 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:39:03.795279 24923 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:39:03.795295 24923 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:39:03.801309 24923 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.100.20:45991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707036041078-13712-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707036041078-13712-0/minicluster-data/master-0/wal
--ranger_kms_url=127.13.100.20:33897/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.13.100.62:35459
--ranger_config_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707036041078-13712-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.62.keytab
--principal=kudu/127.13.100.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.100.62:35459
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.100.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24923
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707036041078-13712-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:39:03.802475 24923 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:39:03.803542 24923 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:39:03.812637 24928 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:39:03.812703 24923 server_base.cc:1034] running on GCE node
W20260929 18:39:03.812642 24929 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:39:03.812666 24931 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:39:03.814486 24923 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:39:03.815778 24923 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:39:03.816972 24923 hybrid_clock.cc:648] HybridClock initialized: now 1790707143816943 us; error 46 us; skew 500 ppm
Sep 29 18:39:03 dist-test-slave-j1xs krb5kdc[24479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707143, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:39:03.820173 24923 init.cc:377] Logged in from keytab as kudu/127.13.100.62@KRBTEST.COM (short username kudu)
I20260929 18:39:03.821290 24923 webserver.cc:460] Webserver started at http://127.13.100.62:40185/ using document root <none> and password file <none>
I20260929 18:39:03.821861 24923 fs_manager.cc:359] Metadata directory not provided
I20260929 18:39:03.822005 24923 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:39:03.822260 24923 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 18:39:03 dist-test-slave-j1xs krb5kdc[24479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707143, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.62@KRBTEST.COM for HTTP/127.13.100.20@KRBTEST.COM
I20260929 18:39:03.877542 24923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707036041078-13712-0/minicluster-data/master-0/data/instance:
uuid: "5b6621b6a9ec4e00873eb77b9d63074a"
format_stamp: "Formatted at 2026-09-29 18:39:03 on dist-test-slave-j1xs"
server_key: "421c332de18e2fd35249b9e34a86f392"
server_key_iv: "a6bc71bcb9d8e1800752c0c4616e2e1d"
server_key_version: "kuduclusterkey@0"
I20260929 18:39:03.878371 24923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707036041078-13712-0/minicluster-data/master-0/wal/instance:
uuid: "5b6621b6a9ec4e00873eb77b9d63074a"
format_stamp: "Formatted at 2026-09-29 18:39:03 on dist-test-slave-j1xs"
server_key: "421c332de18e2fd35249b9e34a86f392"
server_key_iv: "a6bc71bcb9d8e1800752c0c4616e2e1d"
server_key_version: "kuduclusterkey@0"
I20260929 18:39:03.882964 24923 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 18:39:03.898989 24938 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:39:03.900379 24923 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:39:03.900517 24923 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707036041078-13712-0/minicluster-data/master-0/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707036041078-13712-0/minicluster-data/master-0/wal
uuid: "5b6621b6a9ec4e00873eb77b9d63074a"
format_stamp: "Formatted at 2026-09-29 18:39:03 on dist-test-slave-j1xs"
server_key: "421c332de18e2fd35249b9e34a86f392"
server_key_iv: "a6bc71bcb9d8e1800752c0c4616e2e1d"
server_key_version: "kuduclusterkey@0"
I20260929 18:39:03.900643 24923 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707036041078-13712-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707036041078-13712-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707036041078-13712-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:39:03.918418 24923 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:39:03.921134 24923 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:39:03.921315 24923 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:39:03.929239 24923 rpc_server.cc:307] RPC server started. Bound to: 127.13.100.62:35459
I20260929 18:39:03.929303 24990 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.100.62:35459 every 8 connection(s)
I20260929 18:39:03.930177 24923 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
I20260929 18:39:03.932857 24991 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:39:03.935020 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 24923
I20260929 18:39:03.935144 13712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707036041078-13712-0/minicluster-data/master-0/wal/instance
I20260929 18:39:03.935446 13712 external_mini_cluster.cc:1428] Setting key 68361907cba405f9786393c960acd9b8
I20260929 18:39:03.938860 24991 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b6621b6a9ec4e00873eb77b9d63074a: Bootstrap starting.
I20260929 18:39:03.941789 24991 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5b6621b6a9ec4e00873eb77b9d63074a: Neither blocks nor log segments found. Creating new log.
I20260929 18:39:03.942512 24991 log.cc:824] T 00000000000000000000000000000000 P 5b6621b6a9ec4e00873eb77b9d63074a: Log is configured to *not* fsync() on all Append() calls
I20260929 18:39:03.944573 24991 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b6621b6a9ec4e00873eb77b9d63074a: No bootstrap required, opened a new log
Sep 29 18:39:03 dist-test-slave-j1xs krb5kdc[24479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707143, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.100.62@KRBTEST.COM
I20260929 18:39:03.948690 24991 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5b6621b6a9ec4e00873eb77b9d63074a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b6621b6a9ec4e00873eb77b9d63074a" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 35459 } }
I20260929 18:39:03.949025 24991 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5b6621b6a9ec4e00873eb77b9d63074a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:39:03.949091 24991 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5b6621b6a9ec4e00873eb77b9d63074a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b6621b6a9ec4e00873eb77b9d63074a, State: Initialized, Role: FOLLOWER
I20260929 18:39:03.949615 24991 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5b6621b6a9ec4e00873eb77b9d63074a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b6621b6a9ec4e00873eb77b9d63074a" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 35459 } }
I20260929 18:39:03.949821 24991 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5b6621b6a9ec4e00873eb77b9d63074a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:39:03.949922 24991 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5b6621b6a9ec4e00873eb77b9d63074a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:39:03.950064 24991 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5b6621b6a9ec4e00873eb77b9d63074a [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:39:03.951135 24991 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5b6621b6a9ec4e00873eb77b9d63074a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b6621b6a9ec4e00873eb77b9d63074a" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 35459 } }
I20260929 18:39:03.951478 24991 leader_election.cc:302] T 00000000000000000000000000000000 P 5b6621b6a9ec4e00873eb77b9d63074a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5b6621b6a9ec4e00873eb77b9d63074a; no voters:
I20260929 18:39:03.951809 24991 leader_election.cc:288] T 00000000000000000000000000000000 P 5b6621b6a9ec4e00873eb77b9d63074a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:39:03.951872 24996 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5b6621b6a9ec4e00873eb77b9d63074a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:39:03.952009 24994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:03.936875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57972 (local address 127.13.100.62:35459)
0929 18:39:03.937303 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0929 18:39:03.937320 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 18:39:03.937361 (+ 41us) server_negotiation.cc:407] Connection header received
0929 18:39:03.938200 (+ 839us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:39:03.938220 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:39:03.938607 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:39:03.938940 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:39:03.939772 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:03.941630 (+ 1858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:03.942849 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:03.943375 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:03.943537 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:03.946690 (+ 3153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:39:03.946697 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:39:03.946713 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:39:03.946743 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:39:03.948809 (+ 2066us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:39:03.949345 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:39:03.949351 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:39:03.949357 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:39:03.949485 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:39:03.949860 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:39:03.949872 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:39:03.949874 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:39:03.950379 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:39:03.950597 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0929 18:39:03.950839 (+ 242us) server_negotiation.cc:299] Negotiation successful
0929 18:39:03.951108 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":125,"threads_started":1}
I20260929 18:39:03.952270 24996 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5b6621b6a9ec4e00873eb77b9d63074a [term 1 LEADER]: Becoming Leader. State: Replica: 5b6621b6a9ec4e00873eb77b9d63074a, State: Running, Role: LEADER
I20260929 18:39:03.952610 24996 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5b6621b6a9ec4e00873eb77b9d63074a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b6621b6a9ec4e00873eb77b9d63074a" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 35459 } }
I20260929 18:39:03.953078 24991 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5b6621b6a9ec4e00873eb77b9d63074a [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:39:03.953822 24996 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b6621b6a9ec4e00873eb77b9d63074a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5b6621b6a9ec4e00873eb77b9d63074a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b6621b6a9ec4e00873eb77b9d63074a" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 35459 } } }
I20260929 18:39:03.954056 24996 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b6621b6a9ec4e00873eb77b9d63074a [sys.catalog]: This master's current role is: LEADER
I20260929 18:39:03.954290 24996 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b6621b6a9ec4e00873eb77b9d63074a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5b6621b6a9ec4e00873eb77b9d63074a. Latest consensus state: current_term: 1 leader_uuid: "5b6621b6a9ec4e00873eb77b9d63074a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b6621b6a9ec4e00873eb77b9d63074a" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 35459 } } }
I20260929 18:39:03.954437 24996 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b6621b6a9ec4e00873eb77b9d63074a [sys.catalog]: This master's current role is: LEADER
I20260929 18:39:03.954748 25005 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:39:03.957695 25005 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:39:03.958350 24991 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707036041078-13712-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260929 18:39:03.967370 25005 catalog_manager.cc:1347] Generated new cluster ID: 99110ed220044b238608027e3aa25891
I20260929 18:39:03.967504 25005 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:39:03.988758 25005 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:39:03.990106 25005 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:39:04.002220 25005 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5b6621b6a9ec4e00873eb77b9d63074a: Generated new TSK 0
I20260929 18:39:04.003031 25005 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.100.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.100.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.100.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.1.keytab.
Entry for principal kudu/127.13.100.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.1.keytab.
I20260929 18:39:04.702775 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707036041078-13712-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707036041078-13712-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707036041078-13712-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707036041078-13712-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.13.100.20:33897/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.100.1:0
--local_ip_for_outbound_sockets=127.13.100.1
--webserver_interface=127.13.100.1
--webserver_port=0
--tserver_master_addrs=127.13.100.62:35459
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.1.keytab
--principal=kudu/127.13.100.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.100.20:45991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707036041078-13712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707036041078-13712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707036041078-13712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:39:04.854467 25036 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:39:04.854950 25036 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:39:04.855034 25036 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:39:04.861056 25036 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:39:04.861152 25036 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:39:04.861255 25036 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.100.1
I20260929 18:39:04.869419 25036 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.100.20:45991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707036041078-13712-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707036041078-13712-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.13.100.20:33897/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.1.keytab
--principal=kudu/127.13.100.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.100.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707036041078-13712-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.100.1
--webserver_port=0
--tserver_master_addrs=127.13.100.62:35459
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25036
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.100.1
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707036041078-13712-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:39:04.870850 25036 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:39:04.872102 25036 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:39:04.884744 25043 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:39:04.884742 25046 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:39:04.885547 25044 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:39:04.885811 25036 server_base.cc:1034] running on GCE node
I20260929 18:39:04.886233 25036 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:39:04.886878 25036 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:39:04.888060 25036 hybrid_clock.cc:648] HybridClock initialized: now 1790707144888037 us; error 40 us; skew 500 ppm
Sep 29 18:39:04 dist-test-slave-j1xs krb5kdc[24479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707144, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:39:04.891461 25036 init.cc:377] Logged in from keytab as kudu/127.13.100.1@KRBTEST.COM (short username kudu)
I20260929 18:39:04.892540 25036 webserver.cc:460] Webserver started at http://127.13.100.1:45959/ using document root <none> and password file <none>
I20260929 18:39:04.893252 25036 fs_manager.cc:359] Metadata directory not provided
I20260929 18:39:04.893362 25036 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:39:04.893611 25036 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 18:39:04 dist-test-slave-j1xs krb5kdc[24479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707144, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.1@KRBTEST.COM for HTTP/127.13.100.20@KRBTEST.COM
I20260929 18:39:04.910600 25036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707036041078-13712-0/minicluster-data/ts-0/data/instance:
uuid: "f713c3438f124c7b9efb9c96e2e1d06d"
format_stamp: "Formatted at 2026-09-29 18:39:04 on dist-test-slave-j1xs"
server_key: "0621dc7740d2a011983be2424eeff85c"
server_key_iv: "92ed89a267e36e294ff88a35df1abf4f"
server_key_version: "kuduclusterkey@0"
I20260929 18:39:04.911330 25036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707036041078-13712-0/minicluster-data/ts-0/wal/instance:
uuid: "f713c3438f124c7b9efb9c96e2e1d06d"
format_stamp: "Formatted at 2026-09-29 18:39:04 on dist-test-slave-j1xs"
server_key: "0621dc7740d2a011983be2424eeff85c"
server_key_iv: "92ed89a267e36e294ff88a35df1abf4f"
server_key_version: "kuduclusterkey@0"
I20260929 18:39:04.916441 25036 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 18:39:04.945093 25053 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:39:04.946666 25036 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260929 18:39:04.946957 25036 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707036041078-13712-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707036041078-13712-0/minicluster-data/ts-0/wal
uuid: "f713c3438f124c7b9efb9c96e2e1d06d"
format_stamp: "Formatted at 2026-09-29 18:39:04 on dist-test-slave-j1xs"
server_key: "0621dc7740d2a011983be2424eeff85c"
server_key_iv: "92ed89a267e36e294ff88a35df1abf4f"
server_key_version: "kuduclusterkey@0"
I20260929 18:39:04.947099 25036 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707036041078-13712-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707036041078-13712-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707036041078-13712-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:39:04.971207 25036 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:39:04.975031 25036 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:39:04.975288 25036 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:39:04.976054 25036 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:39:04.977151 25036 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:39:04.977257 25036 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:39:04.977401 25036 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:39:04.977486 25036 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:39:04.988862 25036 rpc_server.cc:307] RPC server started. Bound to: 127.13.100.1:37219
I20260929 18:39:04.989784 25036 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707036041078-13712-0/minicluster-data/ts-0/data/info.pb
I20260929 18:39:04.992707 25168 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.100.1:37219 every 8 connection(s)
I20260929 18:39:04.994716 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 25036
I20260929 18:39:04.994840 13712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707036041078-13712-0/minicluster-data/ts-0/wal/instance
I20260929 18:39:04.995115 13712 external_mini_cluster.cc:1428] Setting key 2c0bf65d6af88a3bb211c86864c5d276
Sep 29 18:39:04 dist-test-slave-j1xs krb5kdc[24479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707144, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.1@KRBTEST.COM for kudu/127.13.100.62@KRBTEST.COM
I20260929 18:39:05.010474 25172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:04.992497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:35459 (local address 127.13.100.1:49521)
0929 18:39:04.993576 (+ 1079us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:04.993627 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 18:39:04.994659 (+ 1032us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:39:04.995436 (+ 777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:39:04.995445 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:39:04.995777 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:39:04.996432 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:04.996452 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:04.997618 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:04.997622 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:04.999078 (+ 1456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:04.999089 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:04.999925 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:04.999928 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:05.000117 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:05.000792 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:39:05.000813 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:39:05.002549 (+ 1736us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:39:05.004828 (+ 2279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:39:05.004835 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:39:05.004851 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:39:05.007431 (+ 2580us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:39:05.007751 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:39:05.007755 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:39:05.007757 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:39:05.007938 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:39:05.008448 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:39:05.008455 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:39:05.008783 (+ 328us) client_negotiation.cc:769] Sending connection context
0929 18:39:05.009003 (+ 220us) client_negotiation.cc:240] Negotiation successful
0929 18:39:05.009357 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":864,"thread_start_us":131,"threads_started":1}
I20260929 18:39:05.011377 25171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:04.991968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.1:49521 (local address 127.13.100.62:35459)
0929 18:39:04.993321 (+ 1353us) server_negotiation.cc:206] Beginning negotiation
0929 18:39:04.993329 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 18:39:04.993758 (+ 429us) server_negotiation.cc:407] Connection header received
0929 18:39:04.994979 (+ 1221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:39:04.994983 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:39:04.995040 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:39:04.995127 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:39:04.996600 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:04.997469 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:04.999220 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:04.999594 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:04.999753 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:05.002743 (+ 2990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:39:05.002747 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:39:05.002750 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:39:05.002778 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:39:05.004699 (+ 1921us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:39:05.007552 (+ 2853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:39:05.007556 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:39:05.007559 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:39:05.007650 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:39:05.008047 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:39:05.008051 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:39:05.008053 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:39:05.008275 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:39:05.008393 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0929 18:39:05.011072 (+ 2679us) server_negotiation.cc:299] Negotiation successful
0929 18:39:05.011211 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1266,"thread_start_us":99,"threads_started":1}
I20260929 18:39:05.013248 25169 heartbeater.cc:347] Connected to a master server at 127.13.100.62:35459
I20260929 18:39:05.013489 25169 heartbeater.cc:464] Registering TS with master...
I20260929 18:39:05.014176 25169 heartbeater.cc:511] Master 127.13.100.62:35459 requested a full tablet report, sending...
I20260929 18:39:05.016052 24955 ts_manager.cc:194] Registered new tserver with Master: f713c3438f124c7b9efb9c96e2e1d06d (127.13.100.1:37219)
I20260929 18:39:05.017549 24955 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.100.1@KRBTEST.COM'} at 127.13.100.1:49521
WARNING: no policy specified for kudu/127.13.100.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.100.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.100.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.2.keytab.
Entry for principal kudu/127.13.100.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.2.keytab.
I20260929 18:39:05.047205 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707036041078-13712-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707036041078-13712-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707036041078-13712-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707036041078-13712-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.13.100.20:33897/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.100.2:0
--local_ip_for_outbound_sockets=127.13.100.2
--webserver_interface=127.13.100.2
--webserver_port=0
--tserver_master_addrs=127.13.100.62:35459
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.2.keytab
--principal=kudu/127.13.100.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.100.20:45991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707036041078-13712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707036041078-13712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707036041078-13712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 18:39:05 dist-test-slave-j1xs krb5kdc[24479](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790707145, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260929 18:39:05.206102 25175 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:39:05.206525 25175 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:39:05.206636 25175 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:39:05.212683 25175 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:39:05.212828 25175 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:39:05.212965 25175 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.100.2
I20260929 18:39:05.220871 25175 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.100.20:45991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707036041078-13712-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707036041078-13712-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.13.100.20:33897/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.2.keytab
--principal=kudu/127.13.100.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.100.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707036041078-13712-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.100.2
--webserver_port=0
--tserver_master_addrs=127.13.100.62:35459
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25175
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.100.2
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707036041078-13712-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:39:05.222277 25175 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:39:05.223611 25175 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:39:05.235105 25181 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:39:05.235509 25182 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:39:05.235666 25184 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:39:05.236025 25175 server_base.cc:1034] running on GCE node
I20260929 18:39:05.236486 25175 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:39:05.237264 25175 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:39:05.238457 25175 hybrid_clock.cc:648] HybridClock initialized: now 1790707145238438 us; error 34 us; skew 500 ppm
Sep 29 18:39:05 dist-test-slave-j1xs krb5kdc[24479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707145, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:39:05.242447 25175 init.cc:377] Logged in from keytab as kudu/127.13.100.2@KRBTEST.COM (short username kudu)
I20260929 18:39:05.243520 25175 webserver.cc:460] Webserver started at http://127.13.100.2:36867/ using document root <none> and password file <none>
I20260929 18:39:05.244197 25175 fs_manager.cc:359] Metadata directory not provided
I20260929 18:39:05.244274 25175 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:39:05.244484 25175 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 18:39:05 dist-test-slave-j1xs krb5kdc[24479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707145, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.2@KRBTEST.COM for HTTP/127.13.100.20@KRBTEST.COM
I20260929 18:39:05.271927 25175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707036041078-13712-0/minicluster-data/ts-1/data/instance:
uuid: "34e12ca07dd44ebc9c9021b85a9de88c"
format_stamp: "Formatted at 2026-09-29 18:39:05 on dist-test-slave-j1xs"
server_key: "f10b4c123b37fc253f6c99d7dbae7658"
server_key_iv: "cc6f816a32cdf55adc8429422b3afe3f"
server_key_version: "kuduclusterkey@0"
I20260929 18:39:05.272773 25175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707036041078-13712-0/minicluster-data/ts-1/wal/instance:
uuid: "34e12ca07dd44ebc9c9021b85a9de88c"
format_stamp: "Formatted at 2026-09-29 18:39:05 on dist-test-slave-j1xs"
server_key: "f10b4c123b37fc253f6c99d7dbae7658"
server_key_iv: "cc6f816a32cdf55adc8429422b3afe3f"
server_key_version: "kuduclusterkey@0"
I20260929 18:39:05.294486 25175 fs_manager.cc:691] Time spent creating directory manager: real 0.021s user 0.004s sys 0.000s
I20260929 18:39:05.311337 25191 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:39:05.313300 25175 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260929 18:39:05.313452 25175 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707036041078-13712-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707036041078-13712-0/minicluster-data/ts-1/wal
uuid: "34e12ca07dd44ebc9c9021b85a9de88c"
format_stamp: "Formatted at 2026-09-29 18:39:05 on dist-test-slave-j1xs"
server_key: "f10b4c123b37fc253f6c99d7dbae7658"
server_key_iv: "cc6f816a32cdf55adc8429422b3afe3f"
server_key_version: "kuduclusterkey@0"
I20260929 18:39:05.313597 25175 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707036041078-13712-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707036041078-13712-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707036041078-13712-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:39:05.335700 25175 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:39:05.339610 25175 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:39:05.339884 25175 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:39:05.340677 25175 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:39:05.341820 25175 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:39:05.341954 25175 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:39:05.342063 25175 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:39:05.342139 25175 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:39:05.357471 25175 rpc_server.cc:307] RPC server started. Bound to: 127.13.100.2:36927
I20260929 18:39:05.358898 25175 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707036041078-13712-0/minicluster-data/ts-1/data/info.pb
I20260929 18:39:05.360457 25304 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.100.2:36927 every 8 connection(s)
I20260929 18:39:05.369134 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 25175
I20260929 18:39:05.369269 13712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707036041078-13712-0/minicluster-data/ts-1/wal/instance
I20260929 18:39:05.369560 13712 external_mini_cluster.cc:1428] Setting key db216638111dd60f1546b3fdf1845c72
Sep 29 18:39:05 dist-test-slave-j1xs krb5kdc[24479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707145, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.2@KRBTEST.COM for kudu/127.13.100.62@KRBTEST.COM
I20260929 18:39:05.378554 25171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:05.362591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.2:38083 (local address 127.13.100.62:35459)
0929 18:39:05.362723 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 18:39:05.362728 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:39:05.363646 (+ 918us) server_negotiation.cc:407] Connection header received
0929 18:39:05.364850 (+ 1204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:39:05.364854 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:39:05.364910 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:39:05.364966 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:39:05.366677 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:05.367494 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:05.370423 (+ 2929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:05.370903 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:05.370995 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:05.374217 (+ 3222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:39:05.374221 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:39:05.374223 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:39:05.374251 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:39:05.376047 (+ 1796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:39:05.376717 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:39:05.376721 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:39:05.376723 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:39:05.376806 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:39:05.377208 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:39:05.377211 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:39:05.377213 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:39:05.377438 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:39:05.377534 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 18:39:05.378220 (+ 686us) server_negotiation.cc:299] Negotiation successful
0929 18:39:05.378351 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 18:39:05.380636 25307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:05.363043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:35459 (local address 127.13.100.2:38083)
0929 18:39:05.363496 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:05.363547 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 18:39:05.364600 (+ 1053us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:39:05.365382 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:39:05.365391 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:39:05.365739 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:39:05.366489 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:05.366510 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:05.367658 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:05.367662 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:05.370266 (+ 2604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:05.370278 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:05.371197 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:05.371201 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:05.371402 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:05.372167 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:39:05.372189 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:39:05.374013 (+ 1824us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:39:05.376191 (+ 2178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:39:05.376197 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:39:05.376220 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:39:05.376598 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:39:05.376913 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:39:05.376916 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:39:05.376918 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:39:05.377094 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:39:05.377601 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:39:05.377608 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:39:05.377945 (+ 337us) client_negotiation.cc:769] Sending connection context
0929 18:39:05.379219 (+ 1274us) client_negotiation.cc:240] Negotiation successful
0929 18:39:05.379576 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":132,"threads_started":1}
I20260929 18:39:05.382254 25305 heartbeater.cc:347] Connected to a master server at 127.13.100.62:35459
I20260929 18:39:05.382527 25305 heartbeater.cc:464] Registering TS with master...
I20260929 18:39:05.383044 25305 heartbeater.cc:511] Master 127.13.100.62:35459 requested a full tablet report, sending...
I20260929 18:39:05.384145 24955 ts_manager.cc:194] Registered new tserver with Master: 34e12ca07dd44ebc9c9021b85a9de88c (127.13.100.2:36927)
I20260929 18:39:05.384838 24955 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.100.2@KRBTEST.COM'} at 127.13.100.2:38083
WARNING: no policy specified for kudu/127.13.100.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.100.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.100.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.3.keytab.
Entry for principal kudu/127.13.100.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.3.keytab.
I20260929 18:39:05.418735 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707036041078-13712-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707036041078-13712-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707036041078-13712-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707036041078-13712-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.13.100.20:33897/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.100.3:0
--local_ip_for_outbound_sockets=127.13.100.3
--webserver_interface=127.13.100.3
--webserver_port=0
--tserver_master_addrs=127.13.100.62:35459
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.3.keytab
--principal=kudu/127.13.100.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.100.20:45991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707036041078-13712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707036041078-13712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707036041078-13712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 18:39:05 dist-test-slave-j1xs krb5kdc[24479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707145, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.62@KRBTEST.COM for HTTP/127.13.100.20@KRBTEST.COM
W20260929 18:39:05.619608 25310 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:39:05.620019 25310 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:39:05.620107 25310 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:39:05.626453 25310 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:39:05.626543 25310 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:39:05.626641 25310 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.100.3
I20260929 18:39:05.634801 25310 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.100.20:45991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707036041078-13712-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707036041078-13712-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.13.100.20:33897/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.3.keytab
--principal=kudu/127.13.100.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.100.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707036041078-13712-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.100.3
--webserver_port=0
--tserver_master_addrs=127.13.100.62:35459
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25310
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.100.3
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707036041078-13712-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:39:05.636210 25310 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:39:05.637568 25310 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:39:05.650986 25315 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:39:05.651051 25316 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:39:05.651067 25318 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:39:05.652575 25310 server_base.cc:1034] running on GCE node
I20260929 18:39:05.653137 25310 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:39:05.653857 25310 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:39:05.655071 25310 hybrid_clock.cc:648] HybridClock initialized: now 1790707145655010 us; error 72 us; skew 500 ppm
Sep 29 18:39:05 dist-test-slave-j1xs krb5kdc[24479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707145, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:39:05.658560 25310 init.cc:377] Logged in from keytab as kudu/127.13.100.3@KRBTEST.COM (short username kudu)
I20260929 18:39:05.659664 25310 webserver.cc:460] Webserver started at http://127.13.100.3:39567/ using document root <none> and password file <none>
I20260929 18:39:05.660374 25310 fs_manager.cc:359] Metadata directory not provided
I20260929 18:39:05.660468 25310 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:39:05.660715 25310 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 18:39:05 dist-test-slave-j1xs krb5kdc[24479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707145, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.3@KRBTEST.COM for HTTP/127.13.100.20@KRBTEST.COM
I20260929 18:39:05.686314 25310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707036041078-13712-0/minicluster-data/ts-2/data/instance:
uuid: "6aa028277d764323895f83e4950ed142"
format_stamp: "Formatted at 2026-09-29 18:39:05 on dist-test-slave-j1xs"
server_key: "e29a660daa91c04a33ffdb9f7173b7f1"
server_key_iv: "3e8cec960edd65647386df3b215bc428"
server_key_version: "kuduclusterkey@0"
I20260929 18:39:05.687100 25310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707036041078-13712-0/minicluster-data/ts-2/wal/instance:
uuid: "6aa028277d764323895f83e4950ed142"
format_stamp: "Formatted at 2026-09-29 18:39:05 on dist-test-slave-j1xs"
server_key: "e29a660daa91c04a33ffdb9f7173b7f1"
server_key_iv: "3e8cec960edd65647386df3b215bc428"
server_key_version: "kuduclusterkey@0"
I20260929 18:39:05.692239 25310 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 18:39:05.718366 25325 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:39:05.720104 25310 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.001s sys 0.000s
I20260929 18:39:05.720305 25310 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707036041078-13712-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707036041078-13712-0/minicluster-data/ts-2/wal
uuid: "6aa028277d764323895f83e4950ed142"
format_stamp: "Formatted at 2026-09-29 18:39:05 on dist-test-slave-j1xs"
server_key: "e29a660daa91c04a33ffdb9f7173b7f1"
server_key_iv: "3e8cec960edd65647386df3b215bc428"
server_key_version: "kuduclusterkey@0"
I20260929 18:39:05.720530 25310 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707036041078-13712-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707036041078-13712-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707036041078-13712-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:39:05.740165 25310 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:39:05.747864 25310 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:39:05.748135 25310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:39:05.748822 25310 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:39:05.749912 25310 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:39:05.750026 25310 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:39:05.750134 25310 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:39:05.750202 25310 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:39:05.770759 25310 rpc_server.cc:307] RPC server started. Bound to: 127.13.100.3:34329
I20260929 18:39:05.772011 25310 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707036041078-13712-0/minicluster-data/ts-2/data/info.pb
I20260929 18:39:05.781363 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 25310
I20260929 18:39:05.781518 13712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707036041078-13712-0/minicluster-data/ts-2/wal/instance
I20260929 18:39:05.781804 13712 external_mini_cluster.cc:1428] Setting key c8b04c2780bbea6019d5f1b55b599ddb
I20260929 18:39:05.782989 25438 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.100.3:34329 every 8 connection(s)
Sep 29 18:39:05 dist-test-slave-j1xs krb5kdc[24479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707145, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.3@KRBTEST.COM for kudu/127.13.100.62@KRBTEST.COM
I20260929 18:39:05.814388 25171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:05.786320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.3:46941 (local address 127.13.100.62:35459)
0929 18:39:05.786434 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0929 18:39:05.786439 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:39:05.790393 (+ 3954us) server_negotiation.cc:407] Connection header received
0929 18:39:05.794010 (+ 3617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:39:05.794014 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:39:05.794073 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:39:05.794127 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:39:05.798006 (+ 3879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:05.798849 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:05.803546 (+ 4697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:05.803885 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:05.803993 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:05.807347 (+ 3354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:39:05.807351 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:39:05.807353 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:39:05.807380 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:39:05.809000 (+ 1620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:39:05.809582 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:39:05.809586 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:39:05.809587 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:39:05.809664 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:39:05.810777 (+ 1113us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:39:05.810784 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:39:05.810786 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:39:05.811237 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:39:05.811337 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 18:39:05.814045 (+ 2708us) server_negotiation.cc:299] Negotiation successful
0929 18:39:05.814192 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 18:39:05.816514 25441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:05.785281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:35459 (local address 127.13.100.3:46941)
0929 18:39:05.790223 (+ 4942us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:05.790288 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0929 18:39:05.791826 (+ 1538us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:39:05.794573 (+ 2747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:39:05.794585 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:39:05.794973 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:39:05.795688 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:05.795708 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:05.799016 (+ 3308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:05.799019 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:05.800341 (+ 1322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:05.800352 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:05.804286 (+ 3934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:05.804291 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:05.804583 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:05.805361 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:39:05.805382 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:39:05.807163 (+ 1781us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:39:05.809133 (+ 1970us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:39:05.809138 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:39:05.809154 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:39:05.809456 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:39:05.809761 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:39:05.809763 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:39:05.809765 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:39:05.809957 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:39:05.811413 (+ 1456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:39:05.811421 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:39:05.811813 (+ 392us) client_negotiation.cc:769] Sending connection context
0929 18:39:05.815106 (+ 3293us) client_negotiation.cc:240] Negotiation successful
0929 18:39:05.815470 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4709,"thread_start_us":170,"threads_started":1}
I20260929 18:39:05.825081 25439 heartbeater.cc:347] Connected to a master server at 127.13.100.62:35459
I20260929 18:39:05.825340 25439 heartbeater.cc:464] Registering TS with master...
I20260929 18:39:05.825976 25439 heartbeater.cc:511] Master 127.13.100.62:35459 requested a full tablet report, sending...
I20260929 18:39:05.827121 24955 ts_manager.cc:194] Registered new tserver with Master: 6aa028277d764323895f83e4950ed142 (127.13.100.3:34329)
I20260929 18:39:05.827821 24955 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.100.3@KRBTEST.COM'} at 127.13.100.3:46941
I20260929 18:39:05.830485 13712 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 18:39:05.862757 25171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:05.838080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57988 (local address 127.13.100.62:35459)
0929 18:39:05.838192 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0929 18:39:05.838197 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:39:05.844472 (+ 6275us) server_negotiation.cc:407] Connection header received
0929 18:39:05.844665 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:39:05.844668 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:39:05.844718 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:39:05.844766 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:39:05.845766 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:05.846561 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:05.850338 (+ 3777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:05.850671 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:05.850743 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:05.852627 (+ 1884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:39:05.852631 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:39:05.852634 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:39:05.852660 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:39:05.854351 (+ 1691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:39:05.861355 (+ 7004us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:39:05.861358 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:39:05.861360 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:39:05.861460 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:39:05.861772 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:39:05.861775 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:39:05.861777 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:39:05.862016 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:39:05.862132 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0929 18:39:05.862478 (+ 346us) server_negotiation.cc:299] Negotiation successful
0929 18:39:05.862599 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260929 18:39:05.866520 24955 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57988:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:39:05.870896 24955 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:39:05.901983 25453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:05.890699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.1:37219 (local address 127.0.0.1:46144)
0929 18:39:05.893529 (+ 2830us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:05.893579 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 18:39:05.893711 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:39:05.894629 (+ 918us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:39:05.894632 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:39:05.894650 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:39:05.894767 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:05.894772 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:05.898036 (+ 3264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:05.898040 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:05.899362 (+ 1322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:05.899382 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:05.901437 (+ 2055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:05.901440 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:05.901559 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:05.901599 (+ 40us) client_negotiation.cc:769] Sending connection context
0929 18:39:05.901700 (+ 101us) client_negotiation.cc:240] Negotiation successful
0929 18:39:05.901808 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2671,"mutex_wait_us":137,"thread_start_us":106,"threads_started":1}
I20260929 18:39:05.902338 25455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:05.891647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50216 (local address 127.13.100.2:36927)
0929 18:39:05.892935 (+ 1288us) server_negotiation.cc:206] Beginning negotiation
0929 18:39:05.892942 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 18:39:05.894123 (+ 1181us) server_negotiation.cc:407] Connection header received
0929 18:39:05.894194 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:39:05.894199 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:39:05.894363 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:39:05.894501 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:39:05.896473 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:05.897537 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:05.899343 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:05.900202 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:05.900286 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:05.900433 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0929 18:39:05.902071 (+ 1638us) server_negotiation.cc:299] Negotiation successful
0929 18:39:05.902195 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1182,"thread_start_us":89,"threads_started":1}
I20260929 18:39:05.902808 25454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:05.890936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.2:36927 (local address 127.0.0.1:50216)
0929 18:39:05.893952 (+ 3016us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:05.893963 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 18:39:05.894039 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:39:05.894504 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:39:05.894510 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:39:05.894534 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:39:05.896339 (+ 1805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:05.896347 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:05.897685 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:05.897691 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:05.899201 (+ 1510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:05.899212 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:05.901763 (+ 2551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:05.901766 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:05.901866 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:05.901911 (+ 45us) client_negotiation.cc:769] Sending connection context
0929 18:39:05.902612 (+ 701us) client_negotiation.cc:240] Negotiation successful
0929 18:39:05.902687 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2933,"mutex_wait_us":1,"thread_start_us":74,"threads_started":1}
I20260929 18:39:05.903257 25457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:05.892286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46144 (local address 127.13.100.1:37219)
0929 18:39:05.893291 (+ 1005us) server_negotiation.cc:206] Beginning negotiation
0929 18:39:05.893296 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:39:05.893699 (+ 403us) server_negotiation.cc:407] Connection header received
0929 18:39:05.893923 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:39:05.893929 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:39:05.894156 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:39:05.894311 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:39:05.895035 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:05.896162 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:05.900496 (+ 4334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:05.901299 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:05.901407 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:05.901624 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0929 18:39:05.903018 (+ 1394us) server_negotiation.cc:299] Negotiation successful
0929 18:39:05.903124 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":914,"thread_start_us":112,"threads_started":1}
I20260929 18:39:05.906610 25088 tablet_service.cc:1467] Processing CreateTablet for tablet 0c65b5347d584350985adea76d223e0a (DEFAULT_TABLE table=test-table [id=29705888b1ad4cbc8fe149cb442adf17]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:39:05.906823 25238 tablet_service.cc:1467] Processing CreateTablet for tablet 0c65b5347d584350985adea76d223e0a (DEFAULT_TABLE table=test-table [id=29705888b1ad4cbc8fe149cb442adf17]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:39:05.907745 25088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c65b5347d584350985adea76d223e0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:39:05.907781 25238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c65b5347d584350985adea76d223e0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:39:05.916764 25458 tablet_bootstrap.cc:492] T 0c65b5347d584350985adea76d223e0a P f713c3438f124c7b9efb9c96e2e1d06d: Bootstrap starting.
I20260929 18:39:05.917588 25452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:05.891464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.3:34329 (local address 127.0.0.1:54510)
0929 18:39:05.894953 (+ 3489us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:05.895336 (+ 383us) client_negotiation.cc:174] Beginning negotiation
0929 18:39:05.895487 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:39:05.896030 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:39:05.896033 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:39:05.896047 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:39:05.896124 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:05.896129 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:05.899499 (+ 3370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:05.899502 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:05.901305 (+ 1803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:05.901314 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:05.917170 (+ 15856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:05.917174 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:05.917273 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:05.917296 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 18:39:05.917358 (+ 62us) client_negotiation.cc:240] Negotiation successful
0929 18:39:05.917450 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3436,"thread_start_us":83,"threads_started":1}
I20260929 18:39:05.918102 25456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:05.891966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54510 (local address 127.13.100.3:34329)
0929 18:39:05.893127 (+ 1161us) server_negotiation.cc:206] Beginning negotiation
0929 18:39:05.893137 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 18:39:05.895423 (+ 2286us) server_negotiation.cc:407] Connection header received
0929 18:39:05.895615 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:39:05.895620 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:39:05.895779 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:39:05.895882 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:39:05.896236 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:05.899207 (+ 2971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:05.902653 (+ 3446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:05.915544 (+ 12891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:05.915634 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:05.915765 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0929 18:39:05.917762 (+ 1997us) server_negotiation.cc:299] Negotiation successful
0929 18:39:05.917869 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1054,"thread_start_us":100,"threads_started":1}
I20260929 18:39:05.919987 25458 tablet_bootstrap.cc:654] T 0c65b5347d584350985adea76d223e0a P f713c3438f124c7b9efb9c96e2e1d06d: Neither blocks nor log segments found. Creating new log.
I20260929 18:39:05.920864 25458 log.cc:824] T 0c65b5347d584350985adea76d223e0a P f713c3438f124c7b9efb9c96e2e1d06d: Log is configured to *not* fsync() on all Append() calls
I20260929 18:39:05.920979 25372 tablet_service.cc:1467] Processing CreateTablet for tablet 0c65b5347d584350985adea76d223e0a (DEFAULT_TABLE table=test-table [id=29705888b1ad4cbc8fe149cb442adf17]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:39:05.922223 25459 tablet_bootstrap.cc:492] T 0c65b5347d584350985adea76d223e0a P 34e12ca07dd44ebc9c9021b85a9de88c: Bootstrap starting.
I20260929 18:39:05.922425 25372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c65b5347d584350985adea76d223e0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:39:05.923060 25458 tablet_bootstrap.cc:492] T 0c65b5347d584350985adea76d223e0a P f713c3438f124c7b9efb9c96e2e1d06d: No bootstrap required, opened a new log
I20260929 18:39:05.923264 25458 ts_tablet_manager.cc:1397] T 0c65b5347d584350985adea76d223e0a P f713c3438f124c7b9efb9c96e2e1d06d: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260929 18:39:05.924767 25459 tablet_bootstrap.cc:654] T 0c65b5347d584350985adea76d223e0a P 34e12ca07dd44ebc9c9021b85a9de88c: Neither blocks nor log segments found. Creating new log.
I20260929 18:39:05.925570 25459 log.cc:824] T 0c65b5347d584350985adea76d223e0a P 34e12ca07dd44ebc9c9021b85a9de88c: Log is configured to *not* fsync() on all Append() calls
I20260929 18:39:05.928828 25458 raft_consensus.cc:348] T 0c65b5347d584350985adea76d223e0a P f713c3438f124c7b9efb9c96e2e1d06d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f713c3438f124c7b9efb9c96e2e1d06d" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 37219 } } peers { permanent_uuid: "6aa028277d764323895f83e4950ed142" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 34329 } } peers { permanent_uuid: "34e12ca07dd44ebc9c9021b85a9de88c" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 36927 } }
I20260929 18:39:05.929195 25458 raft_consensus.cc:374] T 0c65b5347d584350985adea76d223e0a P f713c3438f124c7b9efb9c96e2e1d06d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:39:05.929294 25458 raft_consensus.cc:729] T 0c65b5347d584350985adea76d223e0a P f713c3438f124c7b9efb9c96e2e1d06d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f713c3438f124c7b9efb9c96e2e1d06d, State: Initialized, Role: FOLLOWER
I20260929 18:39:05.929814 25458 consensus_queue.cc:260] T 0c65b5347d584350985adea76d223e0a P f713c3438f124c7b9efb9c96e2e1d06d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f713c3438f124c7b9efb9c96e2e1d06d" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 37219 } } peers { permanent_uuid: "6aa028277d764323895f83e4950ed142" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 34329 } } peers { permanent_uuid: "34e12ca07dd44ebc9c9021b85a9de88c" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 36927 } }
I20260929 18:39:05.930922 25458 ts_tablet_manager.cc:1428] T 0c65b5347d584350985adea76d223e0a P f713c3438f124c7b9efb9c96e2e1d06d: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260929 18:39:05.931299 25169 heartbeater.cc:503] Master 127.13.100.62:35459 was elected leader, sending a full tablet report...
I20260929 18:39:05.939831 25459 tablet_bootstrap.cc:492] T 0c65b5347d584350985adea76d223e0a P 34e12ca07dd44ebc9c9021b85a9de88c: No bootstrap required, opened a new log
I20260929 18:39:05.940048 25459 ts_tablet_manager.cc:1397] T 0c65b5347d584350985adea76d223e0a P 34e12ca07dd44ebc9c9021b85a9de88c: Time spent bootstrapping tablet: real 0.018s user 0.000s sys 0.005s
I20260929 18:39:05.944499 25459 raft_consensus.cc:348] T 0c65b5347d584350985adea76d223e0a P 34e12ca07dd44ebc9c9021b85a9de88c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f713c3438f124c7b9efb9c96e2e1d06d" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 37219 } } peers { permanent_uuid: "6aa028277d764323895f83e4950ed142" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 34329 } } peers { permanent_uuid: "34e12ca07dd44ebc9c9021b85a9de88c" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 36927 } }
I20260929 18:39:05.944770 25459 raft_consensus.cc:374] T 0c65b5347d584350985adea76d223e0a P 34e12ca07dd44ebc9c9021b85a9de88c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:39:05.944819 25459 raft_consensus.cc:729] T 0c65b5347d584350985adea76d223e0a P 34e12ca07dd44ebc9c9021b85a9de88c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34e12ca07dd44ebc9c9021b85a9de88c, State: Initialized, Role: FOLLOWER
I20260929 18:39:05.945921 25459 consensus_queue.cc:260] T 0c65b5347d584350985adea76d223e0a P 34e12ca07dd44ebc9c9021b85a9de88c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f713c3438f124c7b9efb9c96e2e1d06d" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 37219 } } peers { permanent_uuid: "6aa028277d764323895f83e4950ed142" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 34329 } } peers { permanent_uuid: "34e12ca07dd44ebc9c9021b85a9de88c" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 36927 } }
I20260929 18:39:05.947037 25459 ts_tablet_manager.cc:1428] T 0c65b5347d584350985adea76d223e0a P 34e12ca07dd44ebc9c9021b85a9de88c: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260929 18:39:05.947434 25305 heartbeater.cc:503] Master 127.13.100.62:35459 was elected leader, sending a full tablet report...
I20260929 18:39:05.958282 25464 tablet_bootstrap.cc:492] T 0c65b5347d584350985adea76d223e0a P 6aa028277d764323895f83e4950ed142: Bootstrap starting.
Sep 29 18:39:05 dist-test-slave-j1xs krb5kdc[24479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707145, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.62@KRBTEST.COM for HTTP/127.13.100.20@KRBTEST.COM
I20260929 18:39:05.961313 25464 tablet_bootstrap.cc:654] T 0c65b5347d584350985adea76d223e0a P 6aa028277d764323895f83e4950ed142: Neither blocks nor log segments found. Creating new log.
I20260929 18:39:05.962240 25464 log.cc:824] T 0c65b5347d584350985adea76d223e0a P 6aa028277d764323895f83e4950ed142: Log is configured to *not* fsync() on all Append() calls
I20260929 18:39:05.979620 25464 tablet_bootstrap.cc:492] T 0c65b5347d584350985adea76d223e0a P 6aa028277d764323895f83e4950ed142: No bootstrap required, opened a new log
I20260929 18:39:05.983636 25464 ts_tablet_manager.cc:1397] T 0c65b5347d584350985adea76d223e0a P 6aa028277d764323895f83e4950ed142: Time spent bootstrapping tablet: real 0.026s user 0.006s sys 0.000s
W20260929 18:39:05.993937 25170 tablet.cc:2369] T 0c65b5347d584350985adea76d223e0a P f713c3438f124c7b9efb9c96e2e1d06d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:39:05.999712 25464 raft_consensus.cc:348] T 0c65b5347d584350985adea76d223e0a P 6aa028277d764323895f83e4950ed142 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f713c3438f124c7b9efb9c96e2e1d06d" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 37219 } } peers { permanent_uuid: "6aa028277d764323895f83e4950ed142" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 34329 } } peers { permanent_uuid: "34e12ca07dd44ebc9c9021b85a9de88c" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 36927 } }
I20260929 18:39:06.001302 25464 raft_consensus.cc:374] T 0c65b5347d584350985adea76d223e0a P 6aa028277d764323895f83e4950ed142 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:39:06.001432 25464 raft_consensus.cc:729] T 0c65b5347d584350985adea76d223e0a P 6aa028277d764323895f83e4950ed142 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6aa028277d764323895f83e4950ed142, State: Initialized, Role: FOLLOWER
I20260929 18:39:06.002636 25464 consensus_queue.cc:260] T 0c65b5347d584350985adea76d223e0a P 6aa028277d764323895f83e4950ed142 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f713c3438f124c7b9efb9c96e2e1d06d" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 37219 } } peers { permanent_uuid: "6aa028277d764323895f83e4950ed142" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 34329 } } peers { permanent_uuid: "34e12ca07dd44ebc9c9021b85a9de88c" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 36927 } }
I20260929 18:39:06.004029 25464 ts_tablet_manager.cc:1428] T 0c65b5347d584350985adea76d223e0a P 6aa028277d764323895f83e4950ed142: Time spent starting tablet: real 0.020s user 0.002s sys 0.004s
I20260929 18:39:06.006390 25439 heartbeater.cc:503] Master 127.13.100.62:35459 was elected leader, sending a full tablet report...
W20260929 18:39:06.037089 25440 tablet.cc:2369] T 0c65b5347d584350985adea76d223e0a P 6aa028277d764323895f83e4950ed142: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:39:06.049294 25462 raft_consensus.cc:482] T 0c65b5347d584350985adea76d223e0a P f713c3438f124c7b9efb9c96e2e1d06d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:39:06.049543 25462 raft_consensus.cc:504] T 0c65b5347d584350985adea76d223e0a P f713c3438f124c7b9efb9c96e2e1d06d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f713c3438f124c7b9efb9c96e2e1d06d" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 37219 } } peers { permanent_uuid: "6aa028277d764323895f83e4950ed142" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 34329 } } peers { permanent_uuid: "34e12ca07dd44ebc9c9021b85a9de88c" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 36927 } }
I20260929 18:39:06.051153 25462 leader_election.cc:288] T 0c65b5347d584350985adea76d223e0a P f713c3438f124c7b9efb9c96e2e1d06d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6aa028277d764323895f83e4950ed142 (127.13.100.3:34329), 34e12ca07dd44ebc9c9021b85a9de88c (127.13.100.2:36927)
I20260929 18:39:06.062520 25456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:06.051928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.1:56033 (local address 127.13.100.3:34329)
0929 18:39:06.052054 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0929 18:39:06.052058 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:39:06.053351 (+ 1293us) server_negotiation.cc:407] Connection header received
0929 18:39:06.053582 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:39:06.053585 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:39:06.053654 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:39:06.053705 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:39:06.055472 (+ 1767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:06.056277 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:06.060596 (+ 4319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:06.061230 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:06.061333 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:06.061388 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0929 18:39:06.062295 (+ 907us) server_negotiation.cc:299] Negotiation successful
0929 18:39:06.062378 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260929 18:39:06.065387 25455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:06.052359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.1:39559 (local address 127.13.100.2:36927)
0929 18:39:06.052507 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0929 18:39:06.052512 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:39:06.057766 (+ 5254us) server_negotiation.cc:407] Connection header received
0929 18:39:06.057995 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:39:06.057999 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:39:06.058052 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:39:06.058103 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:39:06.061852 (+ 3749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:06.062611 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:06.064098 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:06.064714 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:06.064805 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:06.064850 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0929 18:39:06.065182 (+ 332us) server_negotiation.cc:299] Negotiation successful
0929 18:39:06.065260 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 18:39:06.065793 25469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:06.052249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.2:36927 (local address 127.13.100.1:39559)
0929 18:39:06.057669 (+ 5420us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:06.057685 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 18:39:06.057846 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:39:06.058239 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:39:06.058242 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:39:06.058274 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:39:06.058388 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:06.058395 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:06.062755 (+ 4360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:06.062758 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:06.063950 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:06.063961 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:06.064944 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:06.064947 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:06.065050 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:06.065070 (+ 20us) client_negotiation.cc:769] Sending connection context
0929 18:39:06.065552 (+ 482us) client_negotiation.cc:240] Negotiation successful
0929 18:39:06.065631 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5347,"thread_start_us":97,"threads_started":1}
I20260929 18:39:06.066505 25243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c65b5347d584350985adea76d223e0a" candidate_uuid: "f713c3438f124c7b9efb9c96e2e1d06d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34e12ca07dd44ebc9c9021b85a9de88c" is_pre_election: true
I20260929 18:39:06.066857 25243 raft_consensus.cc:2454] T 0c65b5347d584350985adea76d223e0a P 34e12ca07dd44ebc9c9021b85a9de88c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f713c3438f124c7b9efb9c96e2e1d06d in term 0.
I20260929 18:39:06.067411 25057 leader_election.cc:302] T 0c65b5347d584350985adea76d223e0a P f713c3438f124c7b9efb9c96e2e1d06d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34e12ca07dd44ebc9c9021b85a9de88c, f713c3438f124c7b9efb9c96e2e1d06d; no voters:
I20260929 18:39:06.068600 25462 raft_consensus.cc:2781] T 0c65b5347d584350985adea76d223e0a P f713c3438f124c7b9efb9c96e2e1d06d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:39:06.068706 25462 raft_consensus.cc:482] T 0c65b5347d584350985adea76d223e0a P f713c3438f124c7b9efb9c96e2e1d06d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:39:06.068759 25462 raft_consensus.cc:3037] T 0c65b5347d584350985adea76d223e0a P f713c3438f124c7b9efb9c96e2e1d06d [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:39:06.070127 25462 raft_consensus.cc:504] T 0c65b5347d584350985adea76d223e0a P f713c3438f124c7b9efb9c96e2e1d06d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f713c3438f124c7b9efb9c96e2e1d06d" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 37219 } } peers { permanent_uuid: "6aa028277d764323895f83e4950ed142" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 34329 } } peers { permanent_uuid: "34e12ca07dd44ebc9c9021b85a9de88c" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 36927 } }
I20260929 18:39:06.070502 25462 leader_election.cc:288] T 0c65b5347d584350985adea76d223e0a P f713c3438f124c7b9efb9c96e2e1d06d [CANDIDATE]: Term 1 election: Requested vote from peers 6aa028277d764323895f83e4950ed142 (127.13.100.3:34329), 34e12ca07dd44ebc9c9021b85a9de88c (127.13.100.2:36927)
I20260929 18:39:06.070905 25243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c65b5347d584350985adea76d223e0a" candidate_uuid: "f713c3438f124c7b9efb9c96e2e1d06d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34e12ca07dd44ebc9c9021b85a9de88c"
I20260929 18:39:06.071023 25243 raft_consensus.cc:3037] T 0c65b5347d584350985adea76d223e0a P 34e12ca07dd44ebc9c9021b85a9de88c [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:39:06.072242 25243 raft_consensus.cc:2454] T 0c65b5347d584350985adea76d223e0a P 34e12ca07dd44ebc9c9021b85a9de88c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f713c3438f124c7b9efb9c96e2e1d06d in term 1.
I20260929 18:39:06.072751 25057 leader_election.cc:302] T 0c65b5347d584350985adea76d223e0a P f713c3438f124c7b9efb9c96e2e1d06d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34e12ca07dd44ebc9c9021b85a9de88c, f713c3438f124c7b9efb9c96e2e1d06d; no voters:
I20260929 18:39:06.074733 25462 raft_consensus.cc:2781] T 0c65b5347d584350985adea76d223e0a P f713c3438f124c7b9efb9c96e2e1d06d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:39:06.074950 25462 raft_consensus.cc:686] T 0c65b5347d584350985adea76d223e0a P f713c3438f124c7b9efb9c96e2e1d06d [term 1 LEADER]: Becoming Leader. State: Replica: f713c3438f124c7b9efb9c96e2e1d06d, State: Running, Role: LEADER
I20260929 18:39:06.075301 25462 consensus_queue.cc:237] T 0c65b5347d584350985adea76d223e0a P f713c3438f124c7b9efb9c96e2e1d06d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f713c3438f124c7b9efb9c96e2e1d06d" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 37219 } } peers { permanent_uuid: "6aa028277d764323895f83e4950ed142" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 34329 } } peers { permanent_uuid: "34e12ca07dd44ebc9c9021b85a9de88c" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 36927 } }
I20260929 18:39:06.078670 24955 catalog_manager.cc:5495] T 0c65b5347d584350985adea76d223e0a P f713c3438f124c7b9efb9c96e2e1d06d reported cstate change: term changed from 0 to 1, leader changed from <none> to f713c3438f124c7b9efb9c96e2e1d06d (127.13.100.1). New cstate: current_term: 1 leader_uuid: "f713c3438f124c7b9efb9c96e2e1d06d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f713c3438f124c7b9efb9c96e2e1d06d" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 37219 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6aa028277d764323895f83e4950ed142" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 34329 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "34e12ca07dd44ebc9c9021b85a9de88c" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 36927 } health_report { overall_health: UNKNOWN } } }
I20260929 18:39:06.079545 25468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:06.051521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.3:34329 (local address 127.13.100.1:56033)
0929 18:39:06.053260 (+ 1739us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:06.053275 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 18:39:06.053439 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:39:06.055208 (+ 1769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:39:06.055211 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:39:06.055233 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:39:06.055341 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:06.055347 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:06.057153 (+ 1806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:06.057157 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:06.060454 (+ 3297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:06.060465 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:06.062053 (+ 1588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:06.062056 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:06.062149 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:06.062173 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 18:39:06.079274 (+ 17101us) client_negotiation.cc:240] Negotiation successful
0929 18:39:06.079380 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1664,"thread_start_us":118,"threads_started":1}
I20260929 18:39:06.080307 25393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c65b5347d584350985adea76d223e0a" candidate_uuid: "f713c3438f124c7b9efb9c96e2e1d06d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6aa028277d764323895f83e4950ed142" is_pre_election: true
I20260929 18:39:06.080590 25393 raft_consensus.cc:2454] T 0c65b5347d584350985adea76d223e0a P 6aa028277d764323895f83e4950ed142 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f713c3438f124c7b9efb9c96e2e1d06d in term 0.
I20260929 18:39:06.080921 25392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c65b5347d584350985adea76d223e0a" candidate_uuid: "f713c3438f124c7b9efb9c96e2e1d06d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6aa028277d764323895f83e4950ed142"
I20260929 18:39:06.081033 25392 raft_consensus.cc:3037] T 0c65b5347d584350985adea76d223e0a P 6aa028277d764323895f83e4950ed142 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:39:06.082340 25392 raft_consensus.cc:2454] T 0c65b5347d584350985adea76d223e0a P 6aa028277d764323895f83e4950ed142 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f713c3438f124c7b9efb9c96e2e1d06d in term 1.
I20260929 18:39:06.104213 25457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:06.099006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46160 (local address 127.13.100.1:37219)
0929 18:39:06.099150 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0929 18:39:06.099154 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:39:06.099172 (+ 18us) server_negotiation.cc:407] Connection header received
0929 18:39:06.099262 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:39:06.099265 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:39:06.099332 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:39:06.099454 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:39:06.099855 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:06.100642 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:06.101932 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:06.102281 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:06.102385 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:06.102821 (+ 436us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:39:06.103399 (+ 578us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:39:06.103519 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0929 18:39:06.103969 (+ 450us) server_negotiation.cc:299] Negotiation successful
0929 18:39:06.104047 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
W20260929 18:39:06.111543 25306 tablet.cc:2369] T 0c65b5347d584350985adea76d223e0a P 34e12ca07dd44ebc9c9021b85a9de88c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:39:06.120317 25243 raft_consensus.cc:1261] T 0c65b5347d584350985adea76d223e0a P 34e12ca07dd44ebc9c9021b85a9de88c [term 1 FOLLOWER]: Refusing update from remote peer f713c3438f124c7b9efb9c96e2e1d06d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:39:06.121311 25470 consensus_queue.cc:1035] T 0c65b5347d584350985adea76d223e0a P f713c3438f124c7b9efb9c96e2e1d06d [LEADER]: Connected to new peer: Peer: permanent_uuid: "34e12ca07dd44ebc9c9021b85a9de88c" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 36927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:39:06.127858 25392 raft_consensus.cc:1261] T 0c65b5347d584350985adea76d223e0a P 6aa028277d764323895f83e4950ed142 [term 1 FOLLOWER]: Refusing update from remote peer f713c3438f124c7b9efb9c96e2e1d06d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:39:06.128535 25470 consensus_queue.cc:1035] T 0c65b5347d584350985adea76d223e0a P f713c3438f124c7b9efb9c96e2e1d06d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6aa028277d764323895f83e4950ed142" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 34329 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:39:06.143095 25476 mvcc.cc:205] Tried to move back new op lower bound from 7334736470499340288 to 7334736470327492608. Current Snapshot: MvccSnapshot[applied={T|T < 7334736470499340288}]
I20260929 18:39:06.147320 25473 mvcc.cc:205] Tried to move back new op lower bound from 7334736470499340288 to 7334736470327492608. Current Snapshot: MvccSnapshot[applied={T|T < 7334736470499340288}]
I20260929 18:39:06.147817 25474 mvcc.cc:205] Tried to move back new op lower bound from 7334736470499340288 to 7334736470327492608. Current Snapshot: MvccSnapshot[applied={T|T < 7334736470499340288}]
I20260929 18:39:06.157235 24955 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57988:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:39:06.157404 24955 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57988:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:39:06.160096 24955 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5b6621b6a9ec4e00873eb77b9d63074a: Sending DeleteTablet for 3 replicas of tablet 0c65b5347d584350985adea76d223e0a
I20260929 18:39:06.160949 25238 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c65b5347d584350985adea76d223e0a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:39:06 UTC) from {username='kudu', principal='kudu/127.13.100.62@KRBTEST.COM'} at 127.0.0.1:50216
I20260929 18:39:06.161872 25372 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c65b5347d584350985adea76d223e0a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:39:06 UTC) from {username='kudu', principal='kudu/127.13.100.62@KRBTEST.COM'} at 127.0.0.1:54510
I20260929 18:39:06.162485 25088 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c65b5347d584350985adea76d223e0a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:39:06 UTC) from {username='kudu', principal='kudu/127.13.100.62@KRBTEST.COM'} at 127.0.0.1:46144
I20260929 18:39:06.163700 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 25036
I20260929 18:39:06.168859 25483 tablet_replica.cc:330] T 0c65b5347d584350985adea76d223e0a P 6aa028277d764323895f83e4950ed142: stopping tablet replica
I20260929 18:39:06.169268 25483 raft_consensus.cc:2229] T 0c65b5347d584350985adea76d223e0a P 6aa028277d764323895f83e4950ed142 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:39:06.169670 25483 raft_consensus.cc:2258] T 0c65b5347d584350985adea76d223e0a P 6aa028277d764323895f83e4950ed142 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:39:06.169957 25482 tablet_replica.cc:330] T 0c65b5347d584350985adea76d223e0a P 34e12ca07dd44ebc9c9021b85a9de88c: stopping tablet replica
I20260929 18:39:06.170276 25482 raft_consensus.cc:2229] T 0c65b5347d584350985adea76d223e0a P 34e12ca07dd44ebc9c9021b85a9de88c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:39:06.170934 25482 raft_consensus.cc:2258] T 0c65b5347d584350985adea76d223e0a P 34e12ca07dd44ebc9c9021b85a9de88c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:39:06.173623 25483 ts_tablet_manager.cc:1905] T 0c65b5347d584350985adea76d223e0a P 6aa028277d764323895f83e4950ed142: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:39:06.176712 25483 ts_tablet_manager.cc:1918] T 0c65b5347d584350985adea76d223e0a P 6aa028277d764323895f83e4950ed142: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:39:06.176928 25483 log.cc:1196] T 0c65b5347d584350985adea76d223e0a P 6aa028277d764323895f83e4950ed142: Deleting WAL directory at /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707036041078-13712-0/minicluster-data/ts-2/wal/wals/0c65b5347d584350985adea76d223e0a
I20260929 18:39:06.178042 25482 ts_tablet_manager.cc:1905] T 0c65b5347d584350985adea76d223e0a P 34e12ca07dd44ebc9c9021b85a9de88c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:39:06.184599 25482 ts_tablet_manager.cc:1918] T 0c65b5347d584350985adea76d223e0a P 34e12ca07dd44ebc9c9021b85a9de88c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260929 18:39:06.184986 24941 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.100.1:37219 (error 108)
I20260929 18:39:06.185853 25483 ts_tablet_manager.cc:1939] T 0c65b5347d584350985adea76d223e0a P 6aa028277d764323895f83e4950ed142: Deleting consensus metadata
I20260929 18:39:06.186101 25482 log.cc:1196] T 0c65b5347d584350985adea76d223e0a P 34e12ca07dd44ebc9c9021b85a9de88c: Deleting WAL directory at /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707036041078-13712-0/minicluster-data/ts-1/wal/wals/0c65b5347d584350985adea76d223e0a
I20260929 18:39:06.186498 25482 ts_tablet_manager.cc:1939] T 0c65b5347d584350985adea76d223e0a P 34e12ca07dd44ebc9c9021b85a9de88c: Deleting consensus metadata
I20260929 18:39:06.187928 24941 catalog_manager.cc:4841] TS 6aa028277d764323895f83e4950ed142 (127.13.100.3:34329): tablet 0c65b5347d584350985adea76d223e0a (table test-table [id=29705888b1ad4cbc8fe149cb442adf17]) successfully deleted
I20260929 18:39:06.189317 24942 catalog_manager.cc:4841] TS 34e12ca07dd44ebc9c9021b85a9de88c (127.13.100.2:36927): tablet 0c65b5347d584350985adea76d223e0a (table test-table [id=29705888b1ad4cbc8fe149cb442adf17]) successfully deleted
I20260929 18:39:06.190670 25452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:06.190400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.1:37219 (local address 127.0.0.1:46162)
0929 18:39:06.190516 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:06.190587 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.1:37219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260929 18:39:06.190877 24941 catalog_manager.cc:4584] TS f713c3438f124c7b9efb9c96e2e1d06d (127.13.100.1:37219): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0c65b5347d584350985adea76d223e0a: Network error: Client connection negotiation failed: client connection to 127.13.100.1:37219: connect: Connection refused (error 111)
I20260929 18:39:06.193387 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 25175
I20260929 18:39:06.201573 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 25310
I20260929 18:39:06.209546 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 24923
I20260929 18:39:06.216355 13712 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260929 18:39:06.550669 13712 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260929 18:39:06.550833 13712 mini_ranger.cc:66] Stopping Ranger...
I20260929 18:39:06.895272 13712 mini_ranger.cc:68] Stopped Ranger
2026-09-29 18:39:06.895 UTC [24509] LOG: received smart shutdown request
2026-09-29 18:39:06.898 UTC [24509] LOG: background worker "logical replication launcher" (PID 24518) exited with exit code 1
2026-09-29 18:39:06.898 UTC [24513] LOG: shutting down
2026-09-29 18:39:07.050 UTC [24509] LOG: database system is shut down
2026-09-29T18:39:07Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (46242 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:39:07 dist-test-slave-j1xs krb5kdc[25494](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:39:07 dist-test-slave-j1xs krb5kdc[25494](info): set up 2 sockets
Sep 29 18:39:07 dist-test-slave-j1xs krb5kdc[25494](info): commencing operation
krb5kdc: starting...
W20260929 18:39:08.201841 13712 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/krb5kdc/test-user.keytab.
Sep 29 18:39:08 dist-test-slave-j1xs krb5kdc[25494](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707148, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:39:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:39:08Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.13.100.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.13.100.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.13.100.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/krb5kdc/rangeradmin_127.13.100.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.13.100.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/krb5kdc/rangeradmin_127.13.100.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.13.100.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.13.100.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.13.100.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/krb5kdc/rangerlookup_127.13.100.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.13.100.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/krb5kdc/rangerlookup_127.13.100.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.13.100.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.100.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.100.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/krb5kdc/HTTP_127.13.100.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.13.100.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/krb5kdc/HTTP_127.13.100.20@KRBTEST.COM.keytab.
I20260929 18:39:08.368008 13712 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskpp0tHW/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/minicluster-data/postgres -l logfile start
2026-09-29 18:39:10.842 UTC [25524] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-29 18:39:10.842 UTC [25524] LOG: listening on IPv4 address "127.13.100.20", port 44185
2026-09-29 18:39:10.846 UTC [25524] LOG: listening on Unix socket "/tmp/.s.PGSQL.44185"
2026-09-29 18:39:10.867 UTC [25527] LOG: database system was shut down at 2026-09-29 18:39:09 UTC
2026-09-29 18:39:10.871 UTC [25524] LOG: database system is ready to accept connections
I20260929 18:39:11.813719 13712 mini_postgres.cc:96] Postgres bound to 44185
2026-09-29 18:39:11.819 UTC [25535] FATAL: database "slave" does not exist
127.13.100.20:44185 - accepting connections
I20260929 18:39:11.820411 13712 mini_ranger.cc:161] Starting Ranger...
I20260929 18:39:11.834703 13712 mini_ranger.cc:84] Created miniranger Postgres user
I20260929 18:39:12.339414 13712 mini_ranger.cc:87] Created ranger Postgres database
I20260929 18:39:12.339483 13712 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/minicluster-data/ranger-admin
2026-09-29 18:39:12,605 [I] DB FLAVOR :POSTGRES
2026-09-29 18:39:12,605 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 18:39:12,605 [I] Checking connection..
2026-09-29 18:39:12,605 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpp0tHW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskpp0tHW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.100.20:44185/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-29 18:39:12,856 [I] Checking connection passed.
2026-09-29 18:39:12,856 [I] --------- Verifying version history table ---------
2026-09-29 18:39:12,856 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpp0tHW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskpp0tHW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.100.20:44185/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 18:39:13,085 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 18:39:13,085 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpp0tHW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskpp0tHW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.100.20:44185/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 18:39:13,330 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 18:39:13,331 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-29 18:39:13,331 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpp0tHW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskpp0tHW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.100.20:44185/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskpp0tHW/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-29 18:39:13.568 UTC [25599] WARNING: there is no transaction in progress
2026-09-29 18:39:13,575 [I] create_dbversion_catalog.sql file imported successfully
2026-09-29 18:39:13,575 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpp0tHW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskpp0tHW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.100.20:44185/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 18:39:13,822 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-29 18:39:13,822 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 18:39:13,823 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpp0tHW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskpp0tHW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.100.20:44185/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-29 18:39:14,047 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpp0tHW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskpp0tHW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.100.20:44185/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-29 18:39:14,305 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpp0tHW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskpp0tHW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.100.20:44185/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-j1xs.c.gcp-upstream.internal','N') ;"
2026-09-29 18:39:14,550 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpp0tHW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskpp0tHW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.100.20:44185/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 18:39:14,783 [I] Table x_portal_user does not exist in database ranger
2026-09-29 18:39:14,783 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpp0tHW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskpp0tHW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.100.20:44185/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 18:39:15,030 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-29 18:39:15,030 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpp0tHW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskpp0tHW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.100.20:44185/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 18:39:15,284 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpp0tHW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskpp0tHW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.100.20:44185/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 18:39:15,533 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-29 18:39:15,533 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpp0tHW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskpp0tHW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.100.20:44185/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskpp0tHW/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-29 18:39:16.499 UTC [25707] WARNING: there is no transaction in progress
2026-09-29 18:39:16.515 UTC [25707] WARNING: there is no transaction in progress
2026-09-29 18:39:16.530 UTC [25707] WARNING: there is no transaction in progress
2026-09-29 18:39:16.545 UTC [25707] WARNING: there is no transaction in progress
2026-09-29 18:39:16.563 UTC [25707] WARNING: there is no transaction in progress
2026-09-29 18:39:16.583 UTC [25707] WARNING: there is no transaction in progress
2026-09-29 18:39:16.641 UTC [25707] WARNING: there is no transaction in progress
2026-09-29 18:39:16.648 UTC [25707] WARNING: there is no transaction in progress
2026-09-29 18:39:16.662 UTC [25707] WARNING: there is no transaction in progress
2026-09-29 18:39:16.672 UTC [25707] WARNING: there is no transaction in progress
2026-09-29 18:39:16.683 UTC [25707] WARNING: there is no transaction in progress
2026-09-29 18:39:16.697 UTC [25707] WARNING: there is no transaction in progress
2026-09-29 18:39:16.707 UTC [25707] WARNING: there is no transaction in progress
2026-09-29 18:39:16.713 UTC [25707] WARNING: there is no transaction in progress
2026-09-29 18:39:16.720 UTC [25707] WARNING: there is no transaction in progress
2026-09-29 18:39:16.727 UTC [25707] WARNING: there is no transaction in progress
2026-09-29 18:39:17,352 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-29 18:39:17,352 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpp0tHW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskpp0tHW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.100.20:44185/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-29 18:39:17,582 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-29 18:39:17,582 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpp0tHW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskpp0tHW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.100.20:44185/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 18:39:17,826 [I] Table x_portal_user already exists in database 'ranger'
2026-09-29 18:39:17,826 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpp0tHW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskpp0tHW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.100.20:44185/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 18:39:18,049 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-29 18:39:18,049 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpp0tHW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskpp0tHW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.100.20:44185/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 18:39:18,289 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpp0tHW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskpp0tHW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.100.20:44185/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 18:39:18,520 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpp0tHW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskpp0tHW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.100.20:44185/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-j1xs.c.gcp-upstream.internal';"
2026-09-29 18:39:18,738 [I] CORE_DB_SCHEMA import status has been updated
2026-09-29 18:39:18,739 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpp0tHW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskpp0tHW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.100.20:44185/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-29 18:39:18,990 [I] DB_PATCHES have already been applied
I20260929 18:39:18.995693 13712 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/minicluster-data/ranger-admin:/tmp/dist-test-taskpp0tHW/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskpp0tHW/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskpp0tHW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskpp0tHW/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260929 18:39:18.995779 13712 mini_ranger.cc:193] Using host: 127.13.100.20
I20260929 18:39:18.999128 13712 mini_ranger.cc:227] Ranger admin URL: http://127.13.100.20:58813
Sep 29, 2026 6:39:19 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 6:39:19 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 6:39:19 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/minicluster-data/ranger-admin/ews/webapp
Sep 29, 2026 6:39:19 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 29, 2026 6:39:19 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 29, 2026 6:39:19 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/minicluster-data/ranger-admin/ews/webapp].
Sep 29, 2026 6:39:19 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260929 18:39:20.137081 13712 mini_ranger.cc:160] Time spent starting Ranger: real 8.317s user 0.000s sys 0.009s
Sep 29, 2026 6:39:40 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 29, 2026 6:39:40 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Sep 29, 2026 6:39:40 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 29, 2026 6:39:40 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 29, 2026 6:39:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 29, 2026 6:39:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 29, 2026 6:39:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 29, 2026 6:39:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 29, 2026 6:39:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 29, 2026 6:39:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 29, 2026 6:39:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 29, 2026 6:39:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 29, 2026 6:39:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 29, 2026 6:39:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 29, 2026 6:39:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 29, 2026 6:39:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 29, 2026 6:39:40 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 29, 2026 6:39:41 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260929 18:39:41.955909 13712 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.13.100.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.13.100.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.13.100.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/krb5kdc/rangerkms_127.13.100.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.13.100.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/krb5kdc/rangerkms_127.13.100.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.13.100.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.13.100.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.100.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/krb5kdc/HTTP_127.13.100.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.13.100.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/krb5kdc/HTTP_127.13.100.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 29 18:39:42 dist-test-slave-j1xs krb5kdc[25494](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707182, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260929 18:39:42.040205 13712 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260929 18:39:42.049599 13712 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260929 18:39:42.957546 13712 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260929 18:39:42.957634 13712 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/minicluster-data/ranger-kms
I20260929 18:39:42.957653 13712 mini_ranger_kms.cc:227] Using postgres at 127.13.100.20:44185
2026-09-29 18:39:43,130 [I] DB FLAVOR :POSTGRES
2026-09-29 18:39:43,130 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 18:39:43,130 [I] Checking connection
2026-09-29 18:39:43,130 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpp0tHW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.100.20:44185/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-29 18:39:43,402 [I] connection success
2026-09-29 18:39:43,403 [I] --------- Verifying Ranger DB tables ---------
2026-09-29 18:39:43,403 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-29 18:39:43,403 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpp0tHW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.100.20:44185/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-29 18:39:43,657 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-29 18:39:43,657 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 18:39:43,657 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-29 18:39:43,657 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpp0tHW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.100.20:44185/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-29 18:39:43,976 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260929 18:39:44.395159 13712 mini_ranger_kms.cc:310] Created kms service
I20260929 18:39:44.565214 13712 mini_ranger_kms.cc:326] Created kudu user
I20260929 18:39:44.608466 13712 mini_ranger_kms.cc:343] Created rangerkms user
Sep 29 18:39:44 dist-test-slave-j1xs krb5kdc[25494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707182, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.13.100.20@KRBTEST.COM
I20260929 18:39:44.972922 13712 mini_ranger_kms.cc:384] Added ranger policy
I20260929 18:39:44.973074 13712 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/minicluster-data/ranger-kms:/tmp/dist-test-taskpp0tHW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskpp0tHW/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskpp0tHW/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260929 18:39:44.973104 13712 mini_ranger_kms.cc:240] Using host: 127.13.100.20
I20260929 18:39:44.976694 13712 mini_ranger_kms.cc:276] Ranger KMS PID: 25887
I20260929 18:39:44.976773 13712 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.13.100.20:58181
Sep 29, 2026 6:39:46 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 6:39:46 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 6:39:46 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 29, 2026 6:39:46 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 29, 2026 6:39:47 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/minicluster-data/ranger-kms/ews/webapp].
Sep 29, 2026 6:39:47 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260929 18:39:47.285522 13712 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.245s user 0.000s sys 0.154s
I20260929 18:39:47.285635 13712 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260929 18:39:47.285696 13712 mini_ranger_kms.cc:401] 127.13.100.20:58181/kms/v1/keys
Sep 29 18:39:47 dist-test-slave-j1xs krb5kdc[25494](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790707187, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.13.100.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 29 18:39:48 dist-test-slave-j1xs krb5kdc[25494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707187, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.13.100.20@KRBTEST.COM for HTTP/127.13.100.20@KRBTEST.COM
Sep 29 18:39:48 dist-test-slave-j1xs krb5kdc[25494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707187, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.13.100.20@KRBTEST.COM for HTTP/127.13.100.20@KRBTEST.COM
Sep 29 18:39:50 dist-test-slave-j1xs krb5kdc[25494](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707190, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.13.100.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.100.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.100.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.62.keytab.
Entry for principal kudu/127.13.100.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.62.keytab.
I20260929 18:39:50.928903 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.100.62:45365
--webserver_interface=127.13.100.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.62.keytab
--principal=kudu/127.13.100.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.100.20:35899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.100.62:45365
--ranger_config_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.13.100.20:58181/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:39:51.041352 25938 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:39:51.041607 25938 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:39:51.041644 25938 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:39:51.045727 25938 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:39:51.045789 25938 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:39:51.045807 25938 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:39:51.045825 25938 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:39:51.045840 25938 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:39:51.051301 25938 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.100.20:35899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/minicluster-data/master-0/wal
--ranger_kms_url=127.13.100.20:58181/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.13.100.62:45365
--ranger_config_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.62.keytab
--principal=kudu/127.13.100.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.100.62:45365
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.100.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25938
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:39:51.052353 25938 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:39:51.053376 25938 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:39:51.062076 25943 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:39:51.062106 25944 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:39:51.062177 25938 server_base.cc:1034] running on GCE node
W20260929 18:39:51.062083 25946 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:39:51.062772 25938 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:39:51.063855 25938 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:39:51.065034 25938 hybrid_clock.cc:648] HybridClock initialized: now 1790707191065016 us; error 37 us; skew 500 ppm
Sep 29 18:39:51 dist-test-slave-j1xs krb5kdc[25494](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707191, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:39:51.067857 25938 init.cc:377] Logged in from keytab as kudu/127.13.100.62@KRBTEST.COM (short username kudu)
I20260929 18:39:51.068817 25938 webserver.cc:460] Webserver started at http://127.13.100.62:41303/ using document root <none> and password file <none>
I20260929 18:39:51.069358 25938 fs_manager.cc:359] Metadata directory not provided
I20260929 18:39:51.069440 25938 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:39:51.069650 25938 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 18:39:51 dist-test-slave-j1xs krb5kdc[25494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707191, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.62@KRBTEST.COM for HTTP/127.13.100.20@KRBTEST.COM
I20260929 18:39:51.122337 25938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/minicluster-data/master-0/data/instance:
uuid: "5c93b1d41b784959b6ae9e99ba95b92a"
format_stamp: "Formatted at 2026-09-29 18:39:51 on dist-test-slave-j1xs"
server_key: "917b1e8350309979413bf90b59184729"
server_key_iv: "56dddd3fd9a185a4570dc5b6663c2f3e"
server_key_version: "kuduclusterkey@0"
I20260929 18:39:51.122987 25938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/minicluster-data/master-0/wal/instance:
uuid: "5c93b1d41b784959b6ae9e99ba95b92a"
format_stamp: "Formatted at 2026-09-29 18:39:51 on dist-test-slave-j1xs"
server_key: "917b1e8350309979413bf90b59184729"
server_key_iv: "56dddd3fd9a185a4570dc5b6663c2f3e"
server_key_version: "kuduclusterkey@0"
I20260929 18:39:51.126734 25938 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260929 18:39:51.145496 25953 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:39:51.146898 25938 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 18:39:51.147027 25938 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/minicluster-data/master-0/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/minicluster-data/master-0/wal
uuid: "5c93b1d41b784959b6ae9e99ba95b92a"
format_stamp: "Formatted at 2026-09-29 18:39:51 on dist-test-slave-j1xs"
server_key: "917b1e8350309979413bf90b59184729"
server_key_iv: "56dddd3fd9a185a4570dc5b6663c2f3e"
server_key_version: "kuduclusterkey@0"
I20260929 18:39:51.147158 25938 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:39:51.179066 25938 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:39:51.181962 25938 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:39:51.182161 25938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:39:51.190171 25938 rpc_server.cc:307] RPC server started. Bound to: 127.13.100.62:45365
I20260929 18:39:51.190176 26005 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.100.62:45365 every 8 connection(s)
I20260929 18:39:51.191118 25938 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
I20260929 18:39:51.193847 26006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:39:51.196619 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 25938
I20260929 18:39:51.196748 13712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/minicluster-data/master-0/wal/instance
I20260929 18:39:51.197021 13712 external_mini_cluster.cc:1428] Setting key bb5134a97a1ab3536b11d32173326d03
I20260929 18:39:51.200194 26006 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c93b1d41b784959b6ae9e99ba95b92a: Bootstrap starting.
I20260929 18:39:51.202836 26006 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5c93b1d41b784959b6ae9e99ba95b92a: Neither blocks nor log segments found. Creating new log.
I20260929 18:39:51.203604 26006 log.cc:824] T 00000000000000000000000000000000 P 5c93b1d41b784959b6ae9e99ba95b92a: Log is configured to *not* fsync() on all Append() calls
I20260929 18:39:51.205832 26006 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c93b1d41b784959b6ae9e99ba95b92a: No bootstrap required, opened a new log
Sep 29 18:39:51 dist-test-slave-j1xs krb5kdc[25494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707190, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.100.62@KRBTEST.COM
I20260929 18:39:51.210314 26006 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5c93b1d41b784959b6ae9e99ba95b92a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c93b1d41b784959b6ae9e99ba95b92a" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 45365 } }
I20260929 18:39:51.210604 26006 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5c93b1d41b784959b6ae9e99ba95b92a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:39:51.210686 26006 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5c93b1d41b784959b6ae9e99ba95b92a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c93b1d41b784959b6ae9e99ba95b92a, State: Initialized, Role: FOLLOWER
I20260929 18:39:51.211342 26006 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5c93b1d41b784959b6ae9e99ba95b92a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c93b1d41b784959b6ae9e99ba95b92a" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 45365 } }
I20260929 18:39:51.211503 26006 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5c93b1d41b784959b6ae9e99ba95b92a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:39:51.211591 26006 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5c93b1d41b784959b6ae9e99ba95b92a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:39:51.211690 26006 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5c93b1d41b784959b6ae9e99ba95b92a [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:39:51.212332 26009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:51.198531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34780 (local address 127.13.100.62:45365)
0929 18:39:51.198995 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0929 18:39:51.199013 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 18:39:51.199056 (+ 43us) server_negotiation.cc:407] Connection header received
0929 18:39:51.199846 (+ 790us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:39:51.199870 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:39:51.200283 (+ 413us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:39:51.200652 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:39:51.201515 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:51.202959 (+ 1444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:51.204221 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:51.204737 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:51.204924 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:51.207552 (+ 2628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:39:51.207558 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:39:51.207571 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:39:51.207597 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:39:51.209271 (+ 1674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:39:51.209735 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:39:51.209741 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:39:51.209745 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:39:51.209861 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:39:51.210197 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:39:51.210200 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:39:51.210202 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:39:51.210631 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:39:51.210925 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0929 18:39:51.211205 (+ 280us) server_negotiation.cc:299] Negotiation successful
0929 18:39:51.211500 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":140,"threads_started":1}
I20260929 18:39:51.212805 26006 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5c93b1d41b784959b6ae9e99ba95b92a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c93b1d41b784959b6ae9e99ba95b92a" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 45365 } }
I20260929 18:39:51.213167 26006 leader_election.cc:302] T 00000000000000000000000000000000 P 5c93b1d41b784959b6ae9e99ba95b92a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5c93b1d41b784959b6ae9e99ba95b92a; no voters:
I20260929 18:39:51.213495 26006 leader_election.cc:288] T 00000000000000000000000000000000 P 5c93b1d41b784959b6ae9e99ba95b92a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:39:51.213766 26011 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5c93b1d41b784959b6ae9e99ba95b92a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:39:51.214023 26011 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5c93b1d41b784959b6ae9e99ba95b92a [term 1 LEADER]: Becoming Leader. State: Replica: 5c93b1d41b784959b6ae9e99ba95b92a, State: Running, Role: LEADER
I20260929 18:39:51.214331 26011 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5c93b1d41b784959b6ae9e99ba95b92a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c93b1d41b784959b6ae9e99ba95b92a" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 45365 } }
I20260929 18:39:51.214934 26006 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5c93b1d41b784959b6ae9e99ba95b92a [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:39:51.215536 26011 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c93b1d41b784959b6ae9e99ba95b92a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5c93b1d41b784959b6ae9e99ba95b92a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c93b1d41b784959b6ae9e99ba95b92a" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 45365 } } }
I20260929 18:39:51.215730 26011 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c93b1d41b784959b6ae9e99ba95b92a [sys.catalog]: This master's current role is: LEADER
I20260929 18:39:51.215987 26011 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c93b1d41b784959b6ae9e99ba95b92a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5c93b1d41b784959b6ae9e99ba95b92a. Latest consensus state: current_term: 1 leader_uuid: "5c93b1d41b784959b6ae9e99ba95b92a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c93b1d41b784959b6ae9e99ba95b92a" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 45365 } } }
I20260929 18:39:51.216133 26011 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c93b1d41b784959b6ae9e99ba95b92a [sys.catalog]: This master's current role is: LEADER
I20260929 18:39:51.216488 26019 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:39:51.219600 26019 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:39:51.220098 26006 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260929 18:39:51.229138 26019 catalog_manager.cc:1347] Generated new cluster ID: c71d39f4e1da41f2932bb5aa3eb45b40
I20260929 18:39:51.229339 26019 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:39:51.245012 26019 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:39:51.246423 26019 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:39:51.252677 26019 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5c93b1d41b784959b6ae9e99ba95b92a: Generated new TSK 0
I20260929 18:39:51.253543 26019 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.100.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.100.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.100.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.1.keytab.
Entry for principal kudu/127.13.100.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.1.keytab.
I20260929 18:39:51.940711 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.13.100.20:58181/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.100.1:0
--local_ip_for_outbound_sockets=127.13.100.1
--webserver_interface=127.13.100.1
--webserver_port=0
--tserver_master_addrs=127.13.100.62:45365
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.1.keytab
--principal=kudu/127.13.100.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.100.20:35899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:39:52.073676 26051 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:39:52.074085 26051 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:39:52.074186 26051 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:39:52.078619 26051 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:39:52.078696 26051 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:39:52.078780 26051 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.100.1
I20260929 18:39:52.086472 26051 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.100.20:35899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.13.100.20:58181/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.1.keytab
--principal=kudu/127.13.100.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.100.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.100.1
--webserver_port=0
--tserver_master_addrs=127.13.100.62:45365
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26051
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.100.1
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:39:52.087760 26051 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:39:52.089042 26051 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:39:52.099740 26059 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:39:52.099740 26058 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:39:52.099829 26061 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:39:52.100947 26051 server_base.cc:1034] running on GCE node
I20260929 18:39:52.101385 26051 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:39:52.102123 26051 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:39:52.103367 26051 hybrid_clock.cc:648] HybridClock initialized: now 1790707192103338 us; error 42 us; skew 500 ppm
Sep 29 18:39:52 dist-test-slave-j1xs krb5kdc[25494](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707192, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:39:52.106182 26051 init.cc:377] Logged in from keytab as kudu/127.13.100.1@KRBTEST.COM (short username kudu)
I20260929 18:39:52.107221 26051 webserver.cc:460] Webserver started at http://127.13.100.1:41931/ using document root <none> and password file <none>
I20260929 18:39:52.107785 26051 fs_manager.cc:359] Metadata directory not provided
I20260929 18:39:52.107884 26051 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:39:52.108172 26051 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 18:39:52 dist-test-slave-j1xs krb5kdc[25494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707192, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.1@KRBTEST.COM for HTTP/127.13.100.20@KRBTEST.COM
I20260929 18:39:52.125190 26051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/minicluster-data/ts-0/data/instance:
uuid: "58bd123ee5f14926919cf6dd45a975d4"
format_stamp: "Formatted at 2026-09-29 18:39:52 on dist-test-slave-j1xs"
server_key: "2250b4ede782ad64a8cc0da41b2ed836"
server_key_iv: "ec80dd983a1e90648733ccf6abe00fd6"
server_key_version: "kuduclusterkey@0"
I20260929 18:39:52.126215 26051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/minicluster-data/ts-0/wal/instance:
uuid: "58bd123ee5f14926919cf6dd45a975d4"
format_stamp: "Formatted at 2026-09-29 18:39:52 on dist-test-slave-j1xs"
server_key: "2250b4ede782ad64a8cc0da41b2ed836"
server_key_iv: "ec80dd983a1e90648733ccf6abe00fd6"
server_key_version: "kuduclusterkey@0"
I20260929 18:39:52.139514 26051 fs_manager.cc:691] Time spent creating directory manager: real 0.013s user 0.004s sys 0.000s
I20260929 18:39:52.156976 26070 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:39:52.158937 26051 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 18:39:52.159076 26051 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/minicluster-data/ts-0/wal
uuid: "58bd123ee5f14926919cf6dd45a975d4"
format_stamp: "Formatted at 2026-09-29 18:39:52 on dist-test-slave-j1xs"
server_key: "2250b4ede782ad64a8cc0da41b2ed836"
server_key_iv: "ec80dd983a1e90648733ccf6abe00fd6"
server_key_version: "kuduclusterkey@0"
I20260929 18:39:52.159202 26051 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:39:52.224939 26051 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:39:52.228566 26051 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:39:52.228809 26051 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:39:52.229530 26051 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:39:52.230619 26051 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:39:52.230723 26051 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:39:52.230858 26051 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:39:52.230943 26051 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 18:39:52 dist-test-slave-j1xs krb5kdc[25494](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790707192, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:39:52.243011 26051 rpc_server.cc:307] RPC server started. Bound to: 127.13.100.1:40009
I20260929 18:39:52.243248 26183 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.100.1:40009 every 8 connection(s)
I20260929 18:39:52.244096 26051 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/minicluster-data/ts-0/data/info.pb
I20260929 18:39:52.253219 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 26051
I20260929 18:39:52.253366 13712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/minicluster-data/ts-0/wal/instance
I20260929 18:39:52.253657 13712 external_mini_cluster.cc:1428] Setting key 087a9ec7cda8874e82e6278e3104f21c
Sep 29 18:39:52 dist-test-slave-j1xs krb5kdc[25494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707192, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.1@KRBTEST.COM for kudu/127.13.100.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.13.100.2@KRBTEST.COM; defaulting to no policy
I20260929 18:39:52.267804 26186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:52.246121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.1:38759 (local address 127.13.100.62:45365)
0929 18:39:52.249311 (+ 3190us) server_negotiation.cc:206] Beginning negotiation
0929 18:39:52.249323 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 18:39:52.249935 (+ 612us) server_negotiation.cc:407] Connection header received
0929 18:39:52.250986 (+ 1051us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:39:52.250990 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:39:52.251045 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:39:52.251133 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:39:52.252642 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:52.253535 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:52.258257 (+ 4722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:52.258649 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:52.258761 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:52.263169 (+ 4408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:39:52.263174 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:39:52.263176 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:39:52.263203 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:39:52.265172 (+ 1969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:39:52.265798 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:39:52.265802 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:39:52.265804 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:39:52.265924 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:39:52.266338 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:39:52.266344 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:39:52.266361 (+ 17us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:39:52.266616 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:39:52.266728 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 18:39:52.267475 (+ 747us) server_negotiation.cc:299] Negotiation successful
0929 18:39:52.267612 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3082,"thread_start_us":113,"threads_started":1}
I20260929 18:39:52.268879 26187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:52.246953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:45365 (local address 127.13.100.1:38759)
0929 18:39:52.249452 (+ 2499us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:52.249510 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0929 18:39:52.250659 (+ 1149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:39:52.251439 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:39:52.251448 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:39:52.251804 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:39:52.252460 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:52.252479 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:52.253713 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:52.253717 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:52.255119 (+ 1402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:52.255130 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:52.258950 (+ 3820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:52.258955 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:52.259173 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:52.261167 (+ 1994us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:39:52.261192 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:39:52.262962 (+ 1770us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:39:52.265304 (+ 2342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:39:52.265310 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:39:52.265327 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:39:52.265679 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:39:52.266040 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:39:52.266044 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:39:52.266046 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:39:52.266224 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:39:52.266811 (+ 587us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:39:52.266818 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:39:52.267151 (+ 333us) client_negotiation.cc:769] Sending connection context
0929 18:39:52.267374 (+ 223us) client_negotiation.cc:240] Negotiation successful
0929 18:39:52.267741 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2246,"thread_start_us":128,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.100.2@KRBTEST.COM" created.
I20260929 18:39:52.270337 26184 heartbeater.cc:347] Connected to a master server at 127.13.100.62:45365
I20260929 18:39:52.270604 26184 heartbeater.cc:464] Registering TS with master...
I20260929 18:39:52.271181 26184 heartbeater.cc:511] Master 127.13.100.62:45365 requested a full tablet report, sending...
I20260929 18:39:52.272943 25970 ts_manager.cc:194] Registered new tserver with Master: 58bd123ee5f14926919cf6dd45a975d4 (127.13.100.1:40009)
I20260929 18:39:52.274485 25970 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.100.1@KRBTEST.COM'} at 127.13.100.1:38759
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.100.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.2.keytab.
Entry for principal kudu/127.13.100.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.2.keytab.
I20260929 18:39:52.289126 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.13.100.20:58181/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.100.2:0
--local_ip_for_outbound_sockets=127.13.100.2
--webserver_interface=127.13.100.2
--webserver_port=0
--tserver_master_addrs=127.13.100.62:45365
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.2.keytab
--principal=kudu/127.13.100.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.100.20:35899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:39:52.408605 26190 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:39:52.408895 26190 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:39:52.408989 26190 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:39:52.413120 26190 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:39:52.413246 26190 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:39:52.413379 26190 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.100.2
I20260929 18:39:52.420388 26190 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.100.20:35899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.13.100.20:58181/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.2.keytab
--principal=kudu/127.13.100.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.100.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.100.2
--webserver_port=0
--tserver_master_addrs=127.13.100.62:45365
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26190
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.100.2
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:39:52.421761 26190 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:39:52.423111 26190 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:39:52.433333 26196 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:39:52.433354 26199 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:39:52.433616 26197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:39:52.434350 26190 server_base.cc:1034] running on GCE node
I20260929 18:39:52.434737 26190 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:39:52.435420 26190 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:39:52.436662 26190 hybrid_clock.cc:648] HybridClock initialized: now 1790707192436635 us; error 42 us; skew 500 ppm
Sep 29 18:39:52 dist-test-slave-j1xs krb5kdc[25494](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707192, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:39:52.439519 26190 init.cc:377] Logged in from keytab as kudu/127.13.100.2@KRBTEST.COM (short username kudu)
I20260929 18:39:52.440398 26190 webserver.cc:460] Webserver started at http://127.13.100.2:43021/ using document root <none> and password file <none>
I20260929 18:39:52.440955 26190 fs_manager.cc:359] Metadata directory not provided
I20260929 18:39:52.441045 26190 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:39:52.441274 26190 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 18:39:52 dist-test-slave-j1xs krb5kdc[25494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707192, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.2@KRBTEST.COM for HTTP/127.13.100.20@KRBTEST.COM
I20260929 18:39:52.466279 26190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/minicluster-data/ts-1/data/instance:
uuid: "197c331499ce4dfcbcefc644bde81771"
format_stamp: "Formatted at 2026-09-29 18:39:52 on dist-test-slave-j1xs"
server_key: "e7d4d3531d5d35045cab77959bf05d68"
server_key_iv: "7b165385a59bbd6d0f98f8950bfb7799"
server_key_version: "kuduclusterkey@0"
I20260929 18:39:52.467063 26190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/minicluster-data/ts-1/wal/instance:
uuid: "197c331499ce4dfcbcefc644bde81771"
format_stamp: "Formatted at 2026-09-29 18:39:52 on dist-test-slave-j1xs"
server_key: "e7d4d3531d5d35045cab77959bf05d68"
server_key_iv: "7b165385a59bbd6d0f98f8950bfb7799"
server_key_version: "kuduclusterkey@0"
I20260929 18:39:52.472672 26190 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 18:39:52.492079 26206 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:39:52.493826 26190 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 18:39:52.494086 26190 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/minicluster-data/ts-1/wal
uuid: "197c331499ce4dfcbcefc644bde81771"
format_stamp: "Formatted at 2026-09-29 18:39:52 on dist-test-slave-j1xs"
server_key: "e7d4d3531d5d35045cab77959bf05d68"
server_key_iv: "7b165385a59bbd6d0f98f8950bfb7799"
server_key_version: "kuduclusterkey@0"
I20260929 18:39:52.494320 26190 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:39:52.523846 26190 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:39:52.527840 26190 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:39:52.528098 26190 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:39:52.528898 26190 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:39:52.530014 26190 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:39:52.530115 26190 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:39:52.530207 26190 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:39:52.530269 26190 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:39:52.543635 26190 rpc_server.cc:307] RPC server started. Bound to: 127.13.100.2:33813
I20260929 18:39:52.543897 26319 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.100.2:33813 every 8 connection(s)
I20260929 18:39:52.544806 26190 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/minicluster-data/ts-1/data/info.pb
I20260929 18:39:52.551254 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 26190
I20260929 18:39:52.551398 13712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/minicluster-data/ts-1/wal/instance
I20260929 18:39:52.551694 13712 external_mini_cluster.cc:1428] Setting key cdfef97937771f2e76815dbfb1da7742
Sep 29 18:39:52 dist-test-slave-j1xs krb5kdc[25494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707192, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.2@KRBTEST.COM for kudu/127.13.100.62@KRBTEST.COM
I20260929 18:39:52.563457 26186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:52.547252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.2:37391 (local address 127.13.100.62:45365)
0929 18:39:52.547384 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 18:39:52.547387 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:39:52.548453 (+ 1066us) server_negotiation.cc:407] Connection header received
0929 18:39:52.549672 (+ 1219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:39:52.549676 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:39:52.549735 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:39:52.549834 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:39:52.551624 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:52.552466 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:52.554806 (+ 2340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:52.555140 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:52.555219 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:52.558389 (+ 3170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:39:52.558395 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:39:52.558398 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:39:52.558433 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:39:52.560547 (+ 2114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:39:52.561285 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:39:52.561290 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:39:52.561292 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:39:52.561385 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:39:52.561868 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:39:52.561871 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:39:52.561874 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:39:52.562233 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:39:52.562353 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0929 18:39:52.563126 (+ 773us) server_negotiation.cc:299] Negotiation successful
0929 18:39:52.563264 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 18:39:52.564709 26322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:52.547750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:45365 (local address 127.13.100.2:37391)
0929 18:39:52.548281 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:52.548336 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 18:39:52.549398 (+ 1062us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:39:52.550070 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:39:52.550079 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:39:52.550501 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:39:52.551307 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:52.551407 (+ 100us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:52.552905 (+ 1498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:52.552908 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:52.554658 (+ 1750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:52.554670 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:52.555442 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:52.555446 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:52.555662 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:52.556344 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:39:52.556377 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:39:52.558174 (+ 1797us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:39:52.560686 (+ 2512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:39:52.560694 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:39:52.560716 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:39:52.561132 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:39:52.561496 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:39:52.561501 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:39:52.561504 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:39:52.561733 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:39:52.562380 (+ 647us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:39:52.562390 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:39:52.562783 (+ 393us) client_negotiation.cc:769] Sending connection context
0929 18:39:52.563025 (+ 242us) client_negotiation.cc:240] Negotiation successful
0929 18:39:52.563450 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":130,"threads_started":1}
I20260929 18:39:52.566306 26320 heartbeater.cc:347] Connected to a master server at 127.13.100.62:45365
I20260929 18:39:52.566586 26320 heartbeater.cc:464] Registering TS with master...
I20260929 18:39:52.567155 26320 heartbeater.cc:511] Master 127.13.100.62:45365 requested a full tablet report, sending...
I20260929 18:39:52.568296 25970 ts_manager.cc:194] Registered new tserver with Master: 197c331499ce4dfcbcefc644bde81771 (127.13.100.2:33813)
I20260929 18:39:52.569116 25970 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.100.2@KRBTEST.COM'} at 127.13.100.2:37391
WARNING: no policy specified for kudu/127.13.100.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.100.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.100.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.3.keytab.
Entry for principal kudu/127.13.100.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.3.keytab.
I20260929 18:39:52.618494 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.13.100.20:58181/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.100.3:0
--local_ip_for_outbound_sockets=127.13.100.3
--webserver_interface=127.13.100.3
--webserver_port=0
--tserver_master_addrs=127.13.100.62:45365
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.3.keytab
--principal=kudu/127.13.100.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.100.20:35899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 18:39:52 dist-test-slave-j1xs krb5kdc[25494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707192, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.62@KRBTEST.COM for HTTP/127.13.100.20@KRBTEST.COM
W20260929 18:39:52.806023 26325 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:39:52.806367 26325 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:39:52.806427 26325 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:39:52.812523 26325 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:39:52.812603 26325 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:39:52.812685 26325 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.100.3
I20260929 18:39:52.820597 26325 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.100.20:35899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.13.100.20:58181/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.3.keytab
--principal=kudu/127.13.100.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.100.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.100.3
--webserver_port=0
--tserver_master_addrs=127.13.100.62:45365
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26325
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.100.3
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:39:52.821951 26325 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:39:52.823207 26325 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:39:52.835681 26333 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:39:52.835711 26331 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:39:52.836009 26330 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:39:52.836299 26325 server_base.cc:1034] running on GCE node
I20260929 18:39:52.836716 26325 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:39:52.837452 26325 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:39:52.838642 26325 hybrid_clock.cc:648] HybridClock initialized: now 1790707192838580 us; error 73 us; skew 500 ppm
Sep 29 18:39:52 dist-test-slave-j1xs krb5kdc[25494](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707192, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:39:52.842141 26325 init.cc:377] Logged in from keytab as kudu/127.13.100.3@KRBTEST.COM (short username kudu)
I20260929 18:39:52.843158 26325 webserver.cc:460] Webserver started at http://127.13.100.3:34773/ using document root <none> and password file <none>
I20260929 18:39:52.843827 26325 fs_manager.cc:359] Metadata directory not provided
I20260929 18:39:52.843920 26325 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:39:52.844141 26325 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 18:39:52 dist-test-slave-j1xs krb5kdc[25494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707192, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.3@KRBTEST.COM for HTTP/127.13.100.20@KRBTEST.COM
I20260929 18:39:52.870078 26325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/minicluster-data/ts-2/data/instance:
uuid: "a4b0313e18a04b29a30fde03c6f6be8b"
format_stamp: "Formatted at 2026-09-29 18:39:52 on dist-test-slave-j1xs"
server_key: "06c5f622cd51459981f84ce81cb58f08"
server_key_iv: "017337db7193621f7bf08788f630e43c"
server_key_version: "kuduclusterkey@0"
I20260929 18:39:52.870793 26325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/minicluster-data/ts-2/wal/instance:
uuid: "a4b0313e18a04b29a30fde03c6f6be8b"
format_stamp: "Formatted at 2026-09-29 18:39:52 on dist-test-slave-j1xs"
server_key: "06c5f622cd51459981f84ce81cb58f08"
server_key_iv: "017337db7193621f7bf08788f630e43c"
server_key_version: "kuduclusterkey@0"
I20260929 18:39:52.882678 26325 fs_manager.cc:691] Time spent creating directory manager: real 0.011s user 0.003s sys 0.000s
I20260929 18:39:52.905982 26340 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:39:52.907640 26325 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.003s sys 0.000s
I20260929 18:39:52.907773 26325 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/minicluster-data/ts-2/wal
uuid: "a4b0313e18a04b29a30fde03c6f6be8b"
format_stamp: "Formatted at 2026-09-29 18:39:52 on dist-test-slave-j1xs"
server_key: "06c5f622cd51459981f84ce81cb58f08"
server_key_iv: "017337db7193621f7bf08788f630e43c"
server_key_version: "kuduclusterkey@0"
I20260929 18:39:52.907923 26325 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:39:52.935779 26325 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:39:52.939682 26325 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:39:52.939885 26325 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:39:52.940567 26325 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:39:52.941648 26325 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:39:52.941712 26325 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:39:52.941773 26325 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:39:52.941913 26325 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260929 18:39:52.952924 26325 rpc_server.cc:307] RPC server started. Bound to: 127.13.100.3:34011
I20260929 18:39:52.953052 26453 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.100.3:34011 every 8 connection(s)
I20260929 18:39:52.954571 26325 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/minicluster-data/ts-2/data/info.pb
Sep 29 18:39:52 dist-test-slave-j1xs krb5kdc[25494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707192, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.3@KRBTEST.COM for kudu/127.13.100.62@KRBTEST.COM
I20260929 18:39:52.970100 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 26325
I20260929 18:39:52.970225 13712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/minicluster-data/ts-2/wal/instance
I20260929 18:39:52.970546 13712 external_mini_cluster.cc:1428] Setting key 2cefdc08e77b6fb3abd266c2369fa522
I20260929 18:39:52.977936 26186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:52.956810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.3:32911 (local address 127.13.100.62:45365)
0929 18:39:52.959247 (+ 2437us) server_negotiation.cc:206] Beginning negotiation
0929 18:39:52.959252 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:39:52.959271 (+ 19us) server_negotiation.cc:407] Connection header received
0929 18:39:52.961004 (+ 1733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:39:52.961008 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:39:52.961065 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:39:52.961161 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:39:52.963556 (+ 2395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:52.964430 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:52.966584 (+ 2154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:52.966927 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:52.967011 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:52.970283 (+ 3272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:39:52.970288 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:39:52.970290 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:39:52.970321 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:39:52.973739 (+ 3418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:39:52.976013 (+ 2274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:39:52.976017 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:39:52.976019 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:39:52.976117 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:39:52.976569 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:39:52.976573 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:39:52.976575 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:39:52.976803 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:39:52.976890 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 18:39:52.977604 (+ 714us) server_negotiation.cc:299] Negotiation successful
0929 18:39:52.977739 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2338}
I20260929 18:39:52.979640 26456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:52.957285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:45365 (local address 127.13.100.3:32911)
0929 18:39:52.958253 (+ 968us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:52.958348 (+ 95us) client_negotiation.cc:174] Beginning negotiation
0929 18:39:52.960642 (+ 2294us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:39:52.961794 (+ 1152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:39:52.961803 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:39:52.962673 (+ 870us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:39:52.963353 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:52.963373 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:52.964658 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:52.964661 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:52.966445 (+ 1784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:52.966456 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:52.967217 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:52.967220 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:52.967425 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:52.968217 (+ 792us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:39:52.968239 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:39:52.970051 (+ 1812us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:39:52.975385 (+ 5334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:39:52.975392 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:39:52.975413 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:39:52.975880 (+ 467us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:39:52.976233 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:39:52.976236 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:39:52.976238 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:39:52.976443 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:39:52.976960 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:39:52.976969 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:39:52.977309 (+ 340us) client_negotiation.cc:769] Sending connection context
0929 18:39:52.978188 (+ 879us) client_negotiation.cc:240] Negotiation successful
0929 18:39:52.978578 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":145,"threads_started":1}
I20260929 18:39:52.980978 26454 heartbeater.cc:347] Connected to a master server at 127.13.100.62:45365
I20260929 18:39:52.981237 26454 heartbeater.cc:464] Registering TS with master...
I20260929 18:39:52.982012 26454 heartbeater.cc:511] Master 127.13.100.62:45365 requested a full tablet report, sending...
I20260929 18:39:52.983249 25970 ts_manager.cc:194] Registered new tserver with Master: a4b0313e18a04b29a30fde03c6f6be8b (127.13.100.3:34011)
I20260929 18:39:52.984025 25970 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.100.3@KRBTEST.COM'} at 127.13.100.3:32911
I20260929 18:39:52.988063 13712 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 18:39:52.988806 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 26051
I20260929 18:39:52.998451 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 26190
I20260929 18:39:53.007081 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 26325
I20260929 18:39:53.012313 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 25938
I20260929 18:39:53.019635 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.100.62:45365
--webserver_interface=127.13.100.62
--webserver_port=41303
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.62.keytab
--principal=kudu/127.13.100.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.100.20:35899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.100.62:45365
--ranger_config_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.13.100.20:58181/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.13.100.20:58181/kms with env {KRB5CCNAME=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:39:53.126657 26457 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:39:53.126924 26457 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:39:53.126964 26457 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:39:53.133841 26457 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:39:53.133946 26457 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:39:53.133977 26457 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:39:53.133999 26457 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:39:53.134020 26457 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:39:53.141629 26457 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.100.20:35899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.13.100.20:58181/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.13.100.62:45365
--ranger_config_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.62.keytab
--principal=kudu/127.13.100.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.100.62:45365
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.100.62
--webserver_port=41303
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26457
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:39:53.142666 26457 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:39:53.143620 26457 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:39:53.152128 26462 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:39:53.152101 26463 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:39:53.152129 26465 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:39:53.152369 26457 server_base.cc:1034] running on GCE node
I20260929 18:39:53.152952 26457 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:39:53.154132 26457 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:39:53.155314 26457 hybrid_clock.cc:648] HybridClock initialized: now 1790707193155304 us; error 28 us; skew 500 ppm
Sep 29 18:39:53 dist-test-slave-j1xs krb5kdc[25494](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707193, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:39:53.158083 26457 init.cc:377] Logged in from keytab as kudu/127.13.100.62@KRBTEST.COM (short username kudu)
I20260929 18:39:53.159000 26457 webserver.cc:460] Webserver started at http://127.13.100.62:41303/ using document root <none> and password file <none>
I20260929 18:39:53.159521 26457 fs_manager.cc:359] Metadata directory not provided
I20260929 18:39:53.159606 26457 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 18:39:53 dist-test-slave-j1xs krb5kdc[25494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707193, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.62@KRBTEST.COM for HTTP/127.13.100.20@KRBTEST.COM
I20260929 18:39:53.206985 26457 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260929 18:39:53.209193 26473 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:39:53.210559 26457 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:39:53.210692 26457 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/minicluster-data/master-0/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/minicluster-data/master-0/wal
uuid: "5c93b1d41b784959b6ae9e99ba95b92a"
format_stamp: "Formatted at 2026-09-29 18:39:51 on dist-test-slave-j1xs"
server_key: "917b1e8350309979413bf90b59184729"
server_key_iv: "56dddd3fd9a185a4570dc5b6663c2f3e"
server_key_version: "kuduclusterkey@0"
I20260929 18:39:53.211220 26457 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:39:53.232493 26457 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:39:53.235263 26457 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:39:53.235452 26457 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:39:53.242452 26457 rpc_server.cc:307] RPC server started. Bound to: 127.13.100.62:45365
I20260929 18:39:53.242468 26525 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.100.62:45365 every 8 connection(s)
I20260929 18:39:53.243466 26457 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
I20260929 18:39:53.246649 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 26457
I20260929 18:39:53.247083 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.13.100.20:58181/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.100.1:40009
--local_ip_for_outbound_sockets=127.13.100.1
--tserver_master_addrs=127.13.100.62:45365
--webserver_port=41931
--webserver_interface=127.13.100.1
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.1.keytab
--principal=kudu/127.13.100.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.100.20:35899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.13.100.20:58181/kms with env {KRB5CCNAME=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 18:39:53.247575 26526 sys_catalog.cc:263] Verifying existing consensus state
I20260929 18:39:53.249099 26526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c93b1d41b784959b6ae9e99ba95b92a: Bootstrap starting.
I20260929 18:39:53.264863 26526 log.cc:824] T 00000000000000000000000000000000 P 5c93b1d41b784959b6ae9e99ba95b92a: Log is configured to *not* fsync() on all Append() calls
I20260929 18:39:53.270938 26526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c93b1d41b784959b6ae9e99ba95b92a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 18:39:53.271302 26526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c93b1d41b784959b6ae9e99ba95b92a: Bootstrap complete.
I20260929 18:39:53.275235 26526 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5c93b1d41b784959b6ae9e99ba95b92a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c93b1d41b784959b6ae9e99ba95b92a" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 45365 } }
I20260929 18:39:53.275628 26526 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5c93b1d41b784959b6ae9e99ba95b92a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c93b1d41b784959b6ae9e99ba95b92a, State: Initialized, Role: FOLLOWER
I20260929 18:39:53.276087 26526 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5c93b1d41b784959b6ae9e99ba95b92a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c93b1d41b784959b6ae9e99ba95b92a" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 45365 } }
I20260929 18:39:53.276225 26526 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5c93b1d41b784959b6ae9e99ba95b92a [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:39:53.276307 26526 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5c93b1d41b784959b6ae9e99ba95b92a [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:39:53.276415 26526 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5c93b1d41b784959b6ae9e99ba95b92a [term 1 FOLLOWER]: Advancing to term 2
I20260929 18:39:53.278234 26526 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5c93b1d41b784959b6ae9e99ba95b92a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c93b1d41b784959b6ae9e99ba95b92a" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 45365 } }
I20260929 18:39:53.278539 26526 leader_election.cc:302] T 00000000000000000000000000000000 P 5c93b1d41b784959b6ae9e99ba95b92a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5c93b1d41b784959b6ae9e99ba95b92a; no voters:
I20260929 18:39:53.278855 26526 leader_election.cc:288] T 00000000000000000000000000000000 P 5c93b1d41b784959b6ae9e99ba95b92a [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 18:39:53.278935 26530 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5c93b1d41b784959b6ae9e99ba95b92a [term 2 FOLLOWER]: Leader election won for term 2
I20260929 18:39:53.279134 26530 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5c93b1d41b784959b6ae9e99ba95b92a [term 2 LEADER]: Becoming Leader. State: Replica: 5c93b1d41b784959b6ae9e99ba95b92a, State: Running, Role: LEADER
I20260929 18:39:53.279417 26530 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5c93b1d41b784959b6ae9e99ba95b92a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c93b1d41b784959b6ae9e99ba95b92a" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 45365 } }
I20260929 18:39:53.280026 26526 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5c93b1d41b784959b6ae9e99ba95b92a [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:39:53.280288 26530 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c93b1d41b784959b6ae9e99ba95b92a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5c93b1d41b784959b6ae9e99ba95b92a. Latest consensus state: current_term: 2 leader_uuid: "5c93b1d41b784959b6ae9e99ba95b92a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c93b1d41b784959b6ae9e99ba95b92a" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 45365 } } }
I20260929 18:39:53.280413 26530 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c93b1d41b784959b6ae9e99ba95b92a [sys.catalog]: This master's current role is: LEADER
I20260929 18:39:53.280751 26534 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:39:53.280876 26531 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c93b1d41b784959b6ae9e99ba95b92a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "5c93b1d41b784959b6ae9e99ba95b92a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c93b1d41b784959b6ae9e99ba95b92a" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 45365 } } }
I20260929 18:39:53.281052 26531 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c93b1d41b784959b6ae9e99ba95b92a [sys.catalog]: This master's current role is: LEADER
I20260929 18:39:53.283918 26534 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:39:53.284346 26526 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260929 18:39:53.284567 26534 catalog_manager.cc:1259] Loaded cluster ID: c71d39f4e1da41f2932bb5aa3eb45b40
I20260929 18:39:53.284619 26534 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:39:53.286406 26534 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:39:53.286934 26534 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 5c93b1d41b784959b6ae9e99ba95b92a: Loaded TSK: 0
I20260929 18:39:53.287806 26534 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 18:39:53.387807 26528 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:39:53.388157 26528 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:39:53.388226 26528 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:39:53.394256 26528 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:39:53.394335 26528 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:39:53.394428 26528 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.100.1
I20260929 18:39:53.402251 26528 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.100.20:35899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.13.100.20:58181/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.1.keytab
--principal=kudu/127.13.100.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.100.1:40009
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.100.1
--webserver_port=41931
--tserver_master_addrs=127.13.100.62:45365
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26528
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.100.1
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:39:53.403614 26528 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:39:53.404868 26528 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:39:53.417567 26564 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:39:53.418035 26566 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:39:53.418838 26563 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:39:53.419333 26528 server_base.cc:1034] running on GCE node
I20260929 18:39:53.419766 26528 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:39:53.420548 26528 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:39:53.421746 26528 hybrid_clock.cc:648] HybridClock initialized: now 1790707193421725 us; error 38 us; skew 500 ppm
Sep 29 18:39:53 dist-test-slave-j1xs krb5kdc[25494](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707193, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:39:53.428725 26528 init.cc:377] Logged in from keytab as kudu/127.13.100.1@KRBTEST.COM (short username kudu)
I20260929 18:39:53.429807 26528 webserver.cc:460] Webserver started at http://127.13.100.1:41931/ using document root <none> and password file <none>
I20260929 18:39:53.430521 26528 fs_manager.cc:359] Metadata directory not provided
I20260929 18:39:53.430619 26528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 18:39:53 dist-test-slave-j1xs krb5kdc[25494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707193, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.1@KRBTEST.COM for HTTP/127.13.100.20@KRBTEST.COM
I20260929 18:39:53.456730 26528 fs_manager.cc:709] Time spent opening directory manager: real 0.008s user 0.003s sys 0.000s
I20260929 18:39:53.459859 26574 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:39:53.461596 26528 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 18:39:53.461719 26528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/minicluster-data/ts-0/wal
uuid: "58bd123ee5f14926919cf6dd45a975d4"
format_stamp: "Formatted at 2026-09-29 18:39:52 on dist-test-slave-j1xs"
server_key: "2250b4ede782ad64a8cc0da41b2ed836"
server_key_iv: "ec80dd983a1e90648733ccf6abe00fd6"
server_key_version: "kuduclusterkey@0"
I20260929 18:39:53.462528 26528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:39:53.498669 26528 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:39:53.502444 26528 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:39:53.502650 26528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:39:53.503340 26528 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:39:53.504372 26528 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:39:53.504448 26528 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:39:53.504508 26528 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:39:53.504540 26528 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:39:53.515429 26528 rpc_server.cc:307] RPC server started. Bound to: 127.13.100.1:40009
I20260929 18:39:53.516557 26528 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/minicluster-data/ts-0/data/info.pb
I20260929 18:39:53.516952 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 26528
I20260929 18:39:53.517469 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.13.100.20:58181/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.100.2:33813
--local_ip_for_outbound_sockets=127.13.100.2
--tserver_master_addrs=127.13.100.62:45365
--webserver_port=43021
--webserver_interface=127.13.100.2
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.2.keytab
--principal=kudu/127.13.100.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.100.20:35899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.13.100.20:58181/kms with env {KRB5CCNAME=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 18:39:53.517841 26687 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.100.1:40009 every 8 connection(s)
Sep 29 18:39:53 dist-test-slave-j1xs krb5kdc[25494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707193, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.1@KRBTEST.COM for kudu/127.13.100.62@KRBTEST.COM
I20260929 18:39:53.535229 26692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:53.520006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:45365 (local address 127.13.100.1:58831)
0929 18:39:53.520805 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:53.520854 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 18:39:53.521841 (+ 987us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:39:53.523263 (+ 1422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:39:53.523271 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:39:53.523614 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:39:53.524262 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:53.524280 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:53.525717 (+ 1437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:53.525720 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:53.526620 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:53.526629 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:53.527362 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:53.527366 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:53.527555 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:53.528134 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:39:53.528149 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:39:53.529452 (+ 1303us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:39:53.531831 (+ 2379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:39:53.531836 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:39:53.531849 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:39:53.532180 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:39:53.532576 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:39:53.532579 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:39:53.532581 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:39:53.532740 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:39:53.533473 (+ 733us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:39:53.533480 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:39:53.533737 (+ 257us) client_negotiation.cc:769] Sending connection context
0929 18:39:53.534001 (+ 264us) client_negotiation.cc:240] Negotiation successful
0929 18:39:53.534297 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":618,"thread_start_us":109,"threads_started":1}
I20260929 18:39:53.536932 26691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:53.519846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.1:58831 (local address 127.13.100.62:45365)
0929 18:39:53.520547 (+ 701us) server_negotiation.cc:206] Beginning negotiation
0929 18:39:53.520567 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 18:39:53.522173 (+ 1606us) server_negotiation.cc:407] Connection header received
0929 18:39:53.522468 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:39:53.522482 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:39:53.522870 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:39:53.523207 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:39:53.524456 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:53.525569 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:53.526743 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:53.527206 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:53.527376 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:53.529629 (+ 2253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:39:53.529634 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:39:53.529650 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:39:53.529680 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:39:53.531695 (+ 2015us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:39:53.532319 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:39:53.532325 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:39:53.532330 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:39:53.532465 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:39:53.532849 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:39:53.532852 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:39:53.532854 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:39:53.533293 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:39:53.533420 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0929 18:39:53.535816 (+ 2396us) server_negotiation.cc:299] Negotiation successful
0929 18:39:53.536086 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":532,"thread_start_us":122,"threads_started":1}
I20260929 18:39:53.539131 26688 heartbeater.cc:347] Connected to a master server at 127.13.100.62:45365
I20260929 18:39:53.539353 26688 heartbeater.cc:464] Registering TS with master...
I20260929 18:39:53.539868 26688 heartbeater.cc:511] Master 127.13.100.62:45365 requested a full tablet report, sending...
W20260929 18:39:53.541810 26688 heartbeater.cc:644] Failed to heartbeat to 127.13.100.62:45365 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 18:39:53.542063 26688 heartbeater.cc:464] Registering TS with master...
I20260929 18:39:53.542241 26688 heartbeater.cc:511] Master 127.13.100.62:45365 requested a full tablet report, sending...
I20260929 18:39:53.543047 26688 heartbeater.cc:464] Registering TS with master...
I20260929 18:39:53.543239 26688 heartbeater.cc:511] Master 127.13.100.62:45365 requested a full tablet report, sending...
W20260929 18:39:53.543982 26688 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 18:39:53.684250 26690 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:39:53.684633 26690 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:39:53.684708 26690 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:39:53.691016 26690 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:39:53.691097 26690 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:39:53.691191 26690 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.100.2
I20260929 18:39:53.699345 26690 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.100.20:35899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.13.100.20:58181/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.2.keytab
--principal=kudu/127.13.100.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.100.2:33813
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.100.2
--webserver_port=43021
--tserver_master_addrs=127.13.100.62:45365
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26690
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.100.2
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:39:53.700775 26690 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:39:53.702203 26690 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:39:53.715183 26698 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:39:53.715291 26700 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:39:53.715855 26697 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:39:53.716730 26690 server_base.cc:1034] running on GCE node
I20260929 18:39:53.717253 26690 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:39:53.718163 26690 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:39:53.719342 26690 hybrid_clock.cc:648] HybridClock initialized: now 1790707193719324 us; error 39 us; skew 500 ppm
Sep 29 18:39:53 dist-test-slave-j1xs krb5kdc[25494](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707193, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:39:53.722242 26690 init.cc:377] Logged in from keytab as kudu/127.13.100.2@KRBTEST.COM (short username kudu)
I20260929 18:39:53.723269 26690 webserver.cc:460] Webserver started at http://127.13.100.2:43021/ using document root <none> and password file <none>
I20260929 18:39:53.723841 26690 fs_manager.cc:359] Metadata directory not provided
I20260929 18:39:53.724031 26690 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 18:39:53 dist-test-slave-j1xs krb5kdc[25494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707193, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.2@KRBTEST.COM for HTTP/127.13.100.20@KRBTEST.COM
I20260929 18:39:53.752532 26690 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.003s
I20260929 18:39:53.755402 26708 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:39:53.758268 26690 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260929 18:39:53.758402 26690 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/minicluster-data/ts-1/wal
uuid: "197c331499ce4dfcbcefc644bde81771"
format_stamp: "Formatted at 2026-09-29 18:39:52 on dist-test-slave-j1xs"
server_key: "e7d4d3531d5d35045cab77959bf05d68"
server_key_iv: "7b165385a59bbd6d0f98f8950bfb7799"
server_key_version: "kuduclusterkey@0"
I20260929 18:39:53.759148 26690 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:39:53.781738 26690 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:39:53.785300 26690 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:39:53.785516 26690 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:39:53.786263 26690 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:39:53.787281 26690 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:39:53.787359 26690 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:39:53.787420 26690 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:39:53.787468 26690 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:39:53.798909 26690 rpc_server.cc:307] RPC server started. Bound to: 127.13.100.2:33813
I20260929 18:39:53.798982 26822 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.100.2:33813 every 8 connection(s)
I20260929 18:39:53.799912 26690 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/minicluster-data/ts-1/data/info.pb
I20260929 18:39:53.803249 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 26690
I20260929 18:39:53.803730 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.13.100.20:58181/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.100.3:34011
--local_ip_for_outbound_sockets=127.13.100.3
--tserver_master_addrs=127.13.100.62:45365
--webserver_port=34773
--webserver_interface=127.13.100.3
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.3.keytab
--principal=kudu/127.13.100.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.100.20:35899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.13.100.20:58181/kms with env {KRB5CCNAME=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 18:39:53 dist-test-slave-j1xs krb5kdc[25494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707193, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.2@KRBTEST.COM for kudu/127.13.100.62@KRBTEST.COM
I20260929 18:39:53.813328 26691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:53.801767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.2:56549 (local address 127.13.100.62:45365)
0929 18:39:53.801972 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0929 18:39:53.801977 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:39:53.802846 (+ 869us) server_negotiation.cc:407] Connection header received
0929 18:39:53.803817 (+ 971us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:39:53.803820 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:39:53.803882 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:39:53.803983 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:39:53.805092 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:53.805620 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:53.806800 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:53.807032 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:53.807143 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:53.809533 (+ 2390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:39:53.809537 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:39:53.809540 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:39:53.809571 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:39:53.811233 (+ 1662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:39:53.811813 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:39:53.811818 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:39:53.811820 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:39:53.811907 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:39:53.812250 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:39:53.812255 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:39:53.812257 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:39:53.812450 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:39:53.812556 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 18:39:53.813035 (+ 479us) server_negotiation.cc:299] Negotiation successful
0929 18:39:53.813157 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 18:39:53.814334 26825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:53.802201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:45365 (local address 127.13.100.2:56549)
0929 18:39:53.802655 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:53.802706 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 18:39:53.803612 (+ 906us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:39:53.804170 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:39:53.804181 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:39:53.804480 (+ 299us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:39:53.804945 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:53.804958 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:53.805745 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:53.805749 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:53.806675 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:53.806684 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:53.807183 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:53.807187 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:53.807354 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:53.807953 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:39:53.807969 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:39:53.809372 (+ 1403us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:39:53.811364 (+ 1992us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:39:53.811369 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:39:53.811385 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:39:53.811705 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:39:53.812013 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:39:53.812016 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:39:53.812018 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:39:53.812155 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:39:53.812566 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:39:53.812574 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:39:53.812813 (+ 239us) client_negotiation.cc:769] Sending connection context
0929 18:39:53.812998 (+ 185us) client_negotiation.cc:240] Negotiation successful
0929 18:39:53.813306 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":142,"threads_started":1}
I20260929 18:39:53.815498 26823 heartbeater.cc:347] Connected to a master server at 127.13.100.62:45365
I20260929 18:39:53.815744 26823 heartbeater.cc:464] Registering TS with master...
I20260929 18:39:53.816336 26823 heartbeater.cc:511] Master 127.13.100.62:45365 requested a full tablet report, sending...
W20260929 18:39:53.818030 26823 heartbeater.cc:644] Failed to heartbeat to 127.13.100.62:45365 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 18:39:53.818156 26823 heartbeater.cc:464] Registering TS with master...
I20260929 18:39:53.818351 26823 heartbeater.cc:511] Master 127.13.100.62:45365 requested a full tablet report, sending...
I20260929 18:39:53.819134 26823 heartbeater.cc:464] Registering TS with master...
I20260929 18:39:53.819312 26823 heartbeater.cc:511] Master 127.13.100.62:45365 requested a full tablet report, sending...
W20260929 18:39:53.820081 26823 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 18:39:53.942111 26826 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:39:53.942374 26826 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:39:53.942456 26826 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:39:53.946457 26826 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:39:53.946532 26826 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:39:53.946640 26826 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.100.3
I20260929 18:39:53.953634 26826 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.100.20:35899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.13.100.20:58181/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.3.keytab
--principal=kudu/127.13.100.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.100.3:34011
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.100.3
--webserver_port=34773
--tserver_master_addrs=127.13.100.62:45365
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26826
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.100.3
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:39:53.954893 26826 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:39:53.955896 26826 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:39:53.967043 26831 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:39:53.967096 26832 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:39:53.967312 26826 server_base.cc:1034] running on GCE node
W20260929 18:39:53.967041 26834 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:39:53.967963 26826 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:39:53.968755 26826 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:39:53.969966 26826 hybrid_clock.cc:648] HybridClock initialized: now 1790707193969934 us; error 45 us; skew 500 ppm
Sep 29 18:39:53 dist-test-slave-j1xs krb5kdc[25494](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707193, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:39:53.973372 26826 init.cc:377] Logged in from keytab as kudu/127.13.100.3@KRBTEST.COM (short username kudu)
I20260929 18:39:53.974501 26826 webserver.cc:460] Webserver started at http://127.13.100.3:34773/ using document root <none> and password file <none>
I20260929 18:39:53.975194 26826 fs_manager.cc:359] Metadata directory not provided
I20260929 18:39:53.975306 26826 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 18:39:53 dist-test-slave-j1xs krb5kdc[25494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707193, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.3@KRBTEST.COM for HTTP/127.13.100.20@KRBTEST.COM
I20260929 18:39:53.996345 26826 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 18:39:54.002756 26842 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:39:54.004540 26826 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.003s sys 0.000s
I20260929 18:39:54.004729 26826 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/minicluster-data/ts-2/wal
uuid: "a4b0313e18a04b29a30fde03c6f6be8b"
format_stamp: "Formatted at 2026-09-29 18:39:52 on dist-test-slave-j1xs"
server_key: "06c5f622cd51459981f84ce81cb58f08"
server_key_iv: "017337db7193621f7bf08788f630e43c"
server_key_version: "kuduclusterkey@0"
I20260929 18:39:54.005537 26826 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:39:54.030002 26826 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:39:54.033805 26826 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:39:54.034080 26826 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:39:54.034795 26826 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:39:54.035878 26826 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:39:54.035959 26826 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:39:54.036024 26826 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:39:54.036082 26826 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:39:54.046725 26826 rpc_server.cc:307] RPC server started. Bound to: 127.13.100.3:34011
I20260929 18:39:54.048151 26826 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/minicluster-data/ts-2/data/info.pb
I20260929 18:39:54.049400 26955 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.100.3:34011 every 8 connection(s)
I20260929 18:39:54.053099 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 26826
Sep 29 18:39:54 dist-test-slave-j1xs krb5kdc[25494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707193, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.3@KRBTEST.COM for kudu/127.13.100.62@KRBTEST.COM
I20260929 18:39:54.063436 26960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:54.054096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41966 (local address 127.13.100.62:45365)
0929 18:39:54.054338 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0929 18:39:54.054345 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 18:39:54.054355 (+ 10us) server_negotiation.cc:407] Connection header received
0929 18:39:54.054386 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:39:54.054389 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:39:54.054424 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:39:54.054520 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:39:54.055228 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:54.055743 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:54.056636 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:54.056867 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:54.056984 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:54.058521 (+ 1537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:39:54.058524 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:39:54.058527 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:39:54.058558 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:39:54.061522 (+ 2964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:39:54.061986 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:39:54.061990 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:39:54.061992 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:39:54.062075 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:39:54.062354 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:39:54.062357 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:39:54.062359 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:39:54.062540 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:39:54.062988 (+ 448us) server_negotiation.cc:1035] Waiting for connection context
0929 18:39:54.063201 (+ 213us) server_negotiation.cc:299] Negotiation successful
0929 18:39:54.063304 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":74,"threads_started":1}
I20260929 18:39:54.064535 26691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:54.050369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.3:60641 (local address 127.13.100.62:45365)
0929 18:39:54.050529 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0929 18:39:54.050534 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:39:54.051437 (+ 903us) server_negotiation.cc:407] Connection header received
0929 18:39:54.052607 (+ 1170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:39:54.052610 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:39:54.052657 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:39:54.052735 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:39:54.054222 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:54.055041 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:54.056774 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:54.057126 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:54.057224 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:54.060038 (+ 2814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:39:54.060041 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:39:54.060043 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:39:54.060080 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:39:54.061978 (+ 1898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:39:54.062607 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:39:54.062611 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:39:54.062613 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:39:54.062702 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:39:54.063150 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:39:54.063154 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:39:54.063156 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:39:54.063377 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:39:54.063444 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 18:39:54.064281 (+ 837us) server_negotiation.cc:299] Negotiation successful
0929 18:39:54.064394 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 18:39:54.065685 26958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:54.050749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:45365 (local address 127.13.100.3:60641)
0929 18:39:54.051248 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:54.051304 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0929 18:39:54.052355 (+ 1051us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:39:54.052948 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:39:54.052957 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:39:54.053320 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:39:54.054034 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:54.054055 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:54.055216 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:54.055219 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:54.056623 (+ 1404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:54.056634 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:54.057419 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:54.057423 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:54.057621 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:54.058252 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:39:54.058274 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:39:54.059854 (+ 1580us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:39:54.062122 (+ 2268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:39:54.062128 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:39:54.062144 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:39:54.062482 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:39:54.062828 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:39:54.062831 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:39:54.062833 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:39:54.063032 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:39:54.063524 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:39:54.063531 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:39:54.063974 (+ 443us) client_negotiation.cc:769] Sending connection context
0929 18:39:54.064226 (+ 252us) client_negotiation.cc:240] Negotiation successful
0929 18:39:54.064567 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":117,"threads_started":1}
I20260929 18:39:54.066967 26956 heartbeater.cc:347] Connected to a master server at 127.13.100.62:45365
I20260929 18:39:54.067200 26956 heartbeater.cc:464] Registering TS with master...
I20260929 18:39:54.067770 26956 heartbeater.cc:511] Master 127.13.100.62:45365 requested a full tablet report, sending...
W20260929 18:39:54.069479 26956 heartbeater.cc:644] Failed to heartbeat to 127.13.100.62:45365 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 18:39:54.069614 26956 heartbeater.cc:464] Registering TS with master...
I20260929 18:39:54.069788 26956 heartbeater.cc:511] Master 127.13.100.62:45365 requested a full tablet report, sending...
I20260929 18:39:54.070612 26956 heartbeater.cc:464] Registering TS with master...
I20260929 18:39:54.070780 26956 heartbeater.cc:511] Master 127.13.100.62:45365 requested a full tablet report, sending...
W20260929 18:39:54.071507 26956 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 29 18:39:54 dist-test-slave-j1xs krb5kdc[25494](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790707194, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:39:54.544234 26688 heartbeater.cc:464] Registering TS with master...
I20260929 18:39:54.544561 26688 heartbeater.cc:511] Master 127.13.100.62:45365 requested a full tablet report, sending...
I20260929 18:39:54.545634 26490 ts_manager.cc:194] Registered new tserver with Master: 58bd123ee5f14926919cf6dd45a975d4 (127.13.100.1:40009)
I20260929 18:39:54.546955 26490 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.100.1@KRBTEST.COM'} at 127.13.100.1:58831
Sep 29 18:39:54 dist-test-slave-j1xs krb5kdc[25494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707194, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.62@KRBTEST.COM for HTTP/127.13.100.20@KRBTEST.COM
I20260929 18:39:54.820310 26823 heartbeater.cc:464] Registering TS with master...
I20260929 18:39:54.820626 26823 heartbeater.cc:511] Master 127.13.100.62:45365 requested a full tablet report, sending...
I20260929 18:39:54.821456 26490 ts_manager.cc:194] Registered new tserver with Master: 197c331499ce4dfcbcefc644bde81771 (127.13.100.2:33813)
I20260929 18:39:54.822269 26490 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.100.2@KRBTEST.COM'} at 127.13.100.2:56549
Sep 29 18:39:55 dist-test-slave-j1xs krb5kdc[25494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707194, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.62@KRBTEST.COM for HTTP/127.13.100.20@KRBTEST.COM
I20260929 18:39:55.071741 26956 heartbeater.cc:464] Registering TS with master...
I20260929 18:39:55.072052 26956 heartbeater.cc:511] Master 127.13.100.62:45365 requested a full tablet report, sending...
I20260929 18:39:55.072851 26490 ts_manager.cc:194] Registered new tserver with Master: a4b0313e18a04b29a30fde03c6f6be8b (127.13.100.3:34011)
I20260929 18:39:55.073554 26490 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.100.3@KRBTEST.COM'} at 127.13.100.3:60641
I20260929 18:39:55.078013 13712 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 18:39:55.100351 26978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:55.082435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41970 (local address 127.13.100.62:45365)
0929 18:39:55.085592 (+ 3157us) server_negotiation.cc:206] Beginning negotiation
0929 18:39:55.085595 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:39:55.085607 (+ 12us) server_negotiation.cc:407] Connection header received
0929 18:39:55.085644 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:39:55.085646 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:39:55.085683 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:39:55.085792 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:39:55.086985 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:55.087518 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:55.088569 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:55.088833 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:55.088961 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:55.095173 (+ 6212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:39:55.095178 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:39:55.095181 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:39:55.095217 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:39:55.097993 (+ 2776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:39:55.098697 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:39:55.098700 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:39:55.098702 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:39:55.098798 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:39:55.099264 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:39:55.099267 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:39:55.099269 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:39:55.099525 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:39:55.099661 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0929 18:39:55.099979 (+ 318us) server_negotiation.cc:299] Negotiation successful
0929 18:39:55.100132 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3086,"thread_start_us":106,"threads_started":1}
I20260929 18:39:55.104436 26490 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41970:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:39:55.109812 26490 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:39:55.165568 26987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:55.148724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.3:34011 (local address 127.0.0.1:35892)
0929 18:39:55.158245 (+ 9521us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:55.158323 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0929 18:39:55.158514 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:39:55.160348 (+ 1834us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:39:55.160355 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:39:55.160376 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:39:55.160572 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:55.160581 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:55.162121 (+ 1540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:55.162125 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:55.163602 (+ 1477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:55.163613 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:55.164960 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:55.164964 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:55.165082 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:55.165122 (+ 40us) client_negotiation.cc:769] Sending connection context
0929 18:39:55.165239 (+ 117us) client_negotiation.cc:240] Negotiation successful
0929 18:39:55.165375 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9287,"mutex_wait_us":4,"spinlock_wait_cycles":3328,"thread_start_us":108,"threads_started":1}
I20260929 18:39:55.166188 26986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:55.148855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35892 (local address 127.13.100.3:34011)
0929 18:39:55.158157 (+ 9302us) server_negotiation.cc:206] Beginning negotiation
0929 18:39:55.158165 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 18:39:55.159870 (+ 1705us) server_negotiation.cc:407] Connection header received
0929 18:39:55.159963 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:39:55.159968 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:39:55.160183 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:39:55.160337 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:39:55.160730 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:55.161972 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:55.163754 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:55.164613 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:55.164725 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:55.164891 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0929 18:39:55.165937 (+ 1046us) server_negotiation.cc:299] Negotiation successful
0929 18:39:55.166050 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9158,"thread_start_us":79,"threads_started":1}
I20260929 18:39:55.169221 26989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:55.149826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.2:33813 (local address 127.0.0.1:53132)
0929 18:39:55.159308 (+ 9482us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:55.159323 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 18:39:55.159408 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:39:55.160381 (+ 973us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:39:55.160385 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:39:55.160418 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:39:55.160802 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:55.160810 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:55.162276 (+ 1466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:55.162280 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:55.163663 (+ 1383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:55.163672 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:55.168779 (+ 5107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:55.168783 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:55.168878 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:55.168902 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 18:39:55.168966 (+ 64us) client_negotiation.cc:240] Negotiation successful
0929 18:39:55.169078 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9417,"mutex_wait_us":416,"spinlock_wait_cycles":1408,"thread_start_us":114,"threads_started":1}
I20260929 18:39:55.170019 26890 tablet_service.cc:1467] Processing CreateTablet for tablet fe433c14506d423eb8e090418d01f129 (DEFAULT_TABLE table=test-table [id=abf5cd0007794075a5a2fd994df921ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:39:55.171203 26890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe433c14506d423eb8e090418d01f129. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:39:55.171217 26985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:55.150900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.1:40009 (local address 127.0.0.1:41762)
0929 18:39:55.160596 (+ 9696us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:55.160610 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:39:55.162748 (+ 2138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:39:55.164447 (+ 1699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:39:55.164450 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:39:55.164467 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:39:55.164618 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:55.164625 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:55.166179 (+ 1554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:55.166182 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:55.167401 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:55.167411 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:55.170801 (+ 3390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:55.170804 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:55.170916 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:55.170939 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 18:39:55.171002 (+ 63us) client_negotiation.cc:240] Negotiation successful
0929 18:39:55.171085 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9638,"thread_start_us":79,"threads_started":1}
I20260929 18:39:55.171671 26990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:55.151063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41762 (local address 127.13.100.1:40009)
0929 18:39:55.158833 (+ 7770us) server_negotiation.cc:206] Beginning negotiation
0929 18:39:55.158840 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 18:39:55.160702 (+ 1862us) server_negotiation.cc:407] Connection header received
0929 18:39:55.163857 (+ 3155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:39:55.163863 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:39:55.164125 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:39:55.164258 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:39:55.164752 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:55.166037 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:55.169657 (+ 3620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:55.170475 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:55.170574 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:55.170701 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0929 18:39:55.171433 (+ 732us) server_negotiation.cc:299] Negotiation successful
0929 18:39:55.171536 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7654,"thread_start_us":84,"threads_started":1}
I20260929 18:39:55.172133 26988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:55.150241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53132 (local address 127.13.100.2:33813)
0929 18:39:55.159664 (+ 9423us) server_negotiation.cc:206] Beginning negotiation
0929 18:39:55.159671 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 18:39:55.159697 (+ 26us) server_negotiation.cc:407] Connection header received
0929 18:39:55.159776 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:39:55.159782 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:39:55.160043 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:39:55.160213 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:39:55.160935 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:55.162144 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:55.167545 (+ 5401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:55.168452 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:55.168561 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:55.171807 (+ 3246us) server_negotiation.cc:1035] Waiting for connection context
0929 18:39:55.171872 (+ 65us) server_negotiation.cc:299] Negotiation successful
0929 18:39:55.171982 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9309,"thread_start_us":86,"threads_started":1}
I20260929 18:39:55.175045 26750 tablet_service.cc:1467] Processing CreateTablet for tablet fe433c14506d423eb8e090418d01f129 (DEFAULT_TABLE table=test-table [id=abf5cd0007794075a5a2fd994df921ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:39:55.176168 26750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe433c14506d423eb8e090418d01f129. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:39:55.176543 26622 tablet_service.cc:1467] Processing CreateTablet for tablet fe433c14506d423eb8e090418d01f129 (DEFAULT_TABLE table=test-table [id=abf5cd0007794075a5a2fd994df921ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:39:55.177475 26622 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe433c14506d423eb8e090418d01f129. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:39:55.186151 26991 tablet_bootstrap.cc:492] T fe433c14506d423eb8e090418d01f129 P 197c331499ce4dfcbcefc644bde81771: Bootstrap starting.
I20260929 18:39:55.188694 26991 tablet_bootstrap.cc:654] T fe433c14506d423eb8e090418d01f129 P 197c331499ce4dfcbcefc644bde81771: Neither blocks nor log segments found. Creating new log.
I20260929 18:39:55.189708 26991 log.cc:824] T fe433c14506d423eb8e090418d01f129 P 197c331499ce4dfcbcefc644bde81771: Log is configured to *not* fsync() on all Append() calls
I20260929 18:39:55.192139 26991 tablet_bootstrap.cc:492] T fe433c14506d423eb8e090418d01f129 P 197c331499ce4dfcbcefc644bde81771: No bootstrap required, opened a new log
I20260929 18:39:55.192345 26991 ts_tablet_manager.cc:1397] T fe433c14506d423eb8e090418d01f129 P 197c331499ce4dfcbcefc644bde81771: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260929 18:39:55.196614 26991 raft_consensus.cc:348] T fe433c14506d423eb8e090418d01f129 P 197c331499ce4dfcbcefc644bde81771 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b0313e18a04b29a30fde03c6f6be8b" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 34011 } } peers { permanent_uuid: "197c331499ce4dfcbcefc644bde81771" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 33813 } } peers { permanent_uuid: "58bd123ee5f14926919cf6dd45a975d4" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 40009 } }
I20260929 18:39:55.196902 26991 raft_consensus.cc:374] T fe433c14506d423eb8e090418d01f129 P 197c331499ce4dfcbcefc644bde81771 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:39:55.196961 26991 raft_consensus.cc:729] T fe433c14506d423eb8e090418d01f129 P 197c331499ce4dfcbcefc644bde81771 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 197c331499ce4dfcbcefc644bde81771, State: Initialized, Role: FOLLOWER
I20260929 18:39:55.197712 26991 consensus_queue.cc:260] T fe433c14506d423eb8e090418d01f129 P 197c331499ce4dfcbcefc644bde81771 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b0313e18a04b29a30fde03c6f6be8b" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 34011 } } peers { permanent_uuid: "197c331499ce4dfcbcefc644bde81771" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 33813 } } peers { permanent_uuid: "58bd123ee5f14926919cf6dd45a975d4" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 40009 } }
I20260929 18:39:55.198958 26991 ts_tablet_manager.cc:1428] T fe433c14506d423eb8e090418d01f129 P 197c331499ce4dfcbcefc644bde81771: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260929 18:39:55.199380 26823 heartbeater.cc:503] Master 127.13.100.62:45365 was elected leader, sending a full tablet report...
I20260929 18:39:55.206410 26994 tablet_bootstrap.cc:492] T fe433c14506d423eb8e090418d01f129 P 58bd123ee5f14926919cf6dd45a975d4: Bootstrap starting.
I20260929 18:39:55.206914 26992 tablet_bootstrap.cc:492] T fe433c14506d423eb8e090418d01f129 P a4b0313e18a04b29a30fde03c6f6be8b: Bootstrap starting.
I20260929 18:39:55.209630 26992 tablet_bootstrap.cc:654] T fe433c14506d423eb8e090418d01f129 P a4b0313e18a04b29a30fde03c6f6be8b: Neither blocks nor log segments found. Creating new log.
I20260929 18:39:55.213312 26994 tablet_bootstrap.cc:654] T fe433c14506d423eb8e090418d01f129 P 58bd123ee5f14926919cf6dd45a975d4: Neither blocks nor log segments found. Creating new log.
I20260929 18:39:55.214293 26994 log.cc:824] T fe433c14506d423eb8e090418d01f129 P 58bd123ee5f14926919cf6dd45a975d4: Log is configured to *not* fsync() on all Append() calls
I20260929 18:39:55.216614 26992 log.cc:824] T fe433c14506d423eb8e090418d01f129 P a4b0313e18a04b29a30fde03c6f6be8b: Log is configured to *not* fsync() on all Append() calls
I20260929 18:39:55.226985 26992 tablet_bootstrap.cc:492] T fe433c14506d423eb8e090418d01f129 P a4b0313e18a04b29a30fde03c6f6be8b: No bootstrap required, opened a new log
I20260929 18:39:55.227562 26992 ts_tablet_manager.cc:1397] T fe433c14506d423eb8e090418d01f129 P a4b0313e18a04b29a30fde03c6f6be8b: Time spent bootstrapping tablet: real 0.021s user 0.007s sys 0.000s
I20260929 18:39:55.231943 26994 tablet_bootstrap.cc:492] T fe433c14506d423eb8e090418d01f129 P 58bd123ee5f14926919cf6dd45a975d4: No bootstrap required, opened a new log
I20260929 18:39:55.232173 26994 ts_tablet_manager.cc:1397] T fe433c14506d423eb8e090418d01f129 P 58bd123ee5f14926919cf6dd45a975d4: Time spent bootstrapping tablet: real 0.026s user 0.005s sys 0.000s
I20260929 18:39:55.236563 26994 raft_consensus.cc:348] T fe433c14506d423eb8e090418d01f129 P 58bd123ee5f14926919cf6dd45a975d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b0313e18a04b29a30fde03c6f6be8b" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 34011 } } peers { permanent_uuid: "197c331499ce4dfcbcefc644bde81771" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 33813 } } peers { permanent_uuid: "58bd123ee5f14926919cf6dd45a975d4" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 40009 } }
I20260929 18:39:55.241323 26992 raft_consensus.cc:348] T fe433c14506d423eb8e090418d01f129 P a4b0313e18a04b29a30fde03c6f6be8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b0313e18a04b29a30fde03c6f6be8b" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 34011 } } peers { permanent_uuid: "197c331499ce4dfcbcefc644bde81771" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 33813 } } peers { permanent_uuid: "58bd123ee5f14926919cf6dd45a975d4" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 40009 } }
I20260929 18:39:55.241786 26992 raft_consensus.cc:374] T fe433c14506d423eb8e090418d01f129 P a4b0313e18a04b29a30fde03c6f6be8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:39:55.241928 26992 raft_consensus.cc:729] T fe433c14506d423eb8e090418d01f129 P a4b0313e18a04b29a30fde03c6f6be8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4b0313e18a04b29a30fde03c6f6be8b, State: Initialized, Role: FOLLOWER
I20260929 18:39:55.243136 26994 raft_consensus.cc:374] T fe433c14506d423eb8e090418d01f129 P 58bd123ee5f14926919cf6dd45a975d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:39:55.243247 26994 raft_consensus.cc:729] T fe433c14506d423eb8e090418d01f129 P 58bd123ee5f14926919cf6dd45a975d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58bd123ee5f14926919cf6dd45a975d4, State: Initialized, Role: FOLLOWER
I20260929 18:39:55.243734 26994 consensus_queue.cc:260] T fe433c14506d423eb8e090418d01f129 P 58bd123ee5f14926919cf6dd45a975d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b0313e18a04b29a30fde03c6f6be8b" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 34011 } } peers { permanent_uuid: "197c331499ce4dfcbcefc644bde81771" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 33813 } } peers { permanent_uuid: "58bd123ee5f14926919cf6dd45a975d4" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 40009 } }
I20260929 18:39:55.243820 26992 consensus_queue.cc:260] T fe433c14506d423eb8e090418d01f129 P a4b0313e18a04b29a30fde03c6f6be8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b0313e18a04b29a30fde03c6f6be8b" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 34011 } } peers { permanent_uuid: "197c331499ce4dfcbcefc644bde81771" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 33813 } } peers { permanent_uuid: "58bd123ee5f14926919cf6dd45a975d4" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 40009 } }
I20260929 18:39:55.245030 26992 ts_tablet_manager.cc:1428] T fe433c14506d423eb8e090418d01f129 P a4b0313e18a04b29a30fde03c6f6be8b: Time spent starting tablet: real 0.016s user 0.006s sys 0.000s
I20260929 18:39:55.245030 26994 ts_tablet_manager.cc:1428] T fe433c14506d423eb8e090418d01f129 P 58bd123ee5f14926919cf6dd45a975d4: Time spent starting tablet: real 0.013s user 0.007s sys 0.000s
I20260929 18:39:55.245424 26688 heartbeater.cc:503] Master 127.13.100.62:45365 was elected leader, sending a full tablet report...
I20260929 18:39:55.250402 26956 heartbeater.cc:503] Master 127.13.100.62:45365 was elected leader, sending a full tablet report...
I20260929 18:39:55.264698 26995 raft_consensus.cc:482] T fe433c14506d423eb8e090418d01f129 P 197c331499ce4dfcbcefc644bde81771 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:39:55.265235 26995 raft_consensus.cc:504] T fe433c14506d423eb8e090418d01f129 P 197c331499ce4dfcbcefc644bde81771 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b0313e18a04b29a30fde03c6f6be8b" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 34011 } } peers { permanent_uuid: "197c331499ce4dfcbcefc644bde81771" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 33813 } } peers { permanent_uuid: "58bd123ee5f14926919cf6dd45a975d4" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 40009 } }
I20260929 18:39:55.266896 26995 leader_election.cc:288] T fe433c14506d423eb8e090418d01f129 P 197c331499ce4dfcbcefc644bde81771 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a4b0313e18a04b29a30fde03c6f6be8b (127.13.100.3:34011), 58bd123ee5f14926919cf6dd45a975d4 (127.13.100.1:40009)
W20260929 18:39:55.270713 26689 tablet.cc:2369] T fe433c14506d423eb8e090418d01f129 P 58bd123ee5f14926919cf6dd45a975d4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:39:55.277873 26990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:55.268944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.2:34043 (local address 127.13.100.1:40009)
0929 18:39:55.269183 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0929 18:39:55.269188 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:39:55.273061 (+ 3873us) server_negotiation.cc:407] Connection header received
0929 18:39:55.273221 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:39:55.273225 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:39:55.273269 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:39:55.273313 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:39:55.273692 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:55.274490 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:55.276125 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:55.276718 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:55.276816 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:55.276868 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0929 18:39:55.277645 (+ 777us) server_negotiation.cc:299] Negotiation successful
0929 18:39:55.277733 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170}
I20260929 18:39:55.278582 27001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:55.267977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.1:40009 (local address 127.13.100.2:34043)
0929 18:39:55.272988 (+ 5011us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:55.273000 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:39:55.273115 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:39:55.273437 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:39:55.273439 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:39:55.273454 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:39:55.273576 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:55.273581 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:55.274646 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:55.274650 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:55.275979 (+ 1329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:55.275991 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:55.276961 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:55.276964 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:55.277065 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:55.277086 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 18:39:55.278339 (+ 1253us) client_negotiation.cc:240] Negotiation successful
0929 18:39:55.278423 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4962,"thread_start_us":89,"threads_started":1}
I20260929 18:39:55.279359 26642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe433c14506d423eb8e090418d01f129" candidate_uuid: "197c331499ce4dfcbcefc644bde81771" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58bd123ee5f14926919cf6dd45a975d4" is_pre_election: true
I20260929 18:39:55.279706 26642 raft_consensus.cc:2454] T fe433c14506d423eb8e090418d01f129 P 58bd123ee5f14926919cf6dd45a975d4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197c331499ce4dfcbcefc644bde81771 in term 0.
I20260929 18:39:55.280233 26712 leader_election.cc:302] T fe433c14506d423eb8e090418d01f129 P 197c331499ce4dfcbcefc644bde81771 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197c331499ce4dfcbcefc644bde81771, 58bd123ee5f14926919cf6dd45a975d4; no voters:
I20260929 18:39:55.281162 26995 raft_consensus.cc:2781] T fe433c14506d423eb8e090418d01f129 P 197c331499ce4dfcbcefc644bde81771 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:39:55.281273 26995 raft_consensus.cc:482] T fe433c14506d423eb8e090418d01f129 P 197c331499ce4dfcbcefc644bde81771 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:39:55.281319 26995 raft_consensus.cc:3037] T fe433c14506d423eb8e090418d01f129 P 197c331499ce4dfcbcefc644bde81771 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:39:55.282894 26995 raft_consensus.cc:504] T fe433c14506d423eb8e090418d01f129 P 197c331499ce4dfcbcefc644bde81771 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b0313e18a04b29a30fde03c6f6be8b" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 34011 } } peers { permanent_uuid: "197c331499ce4dfcbcefc644bde81771" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 33813 } } peers { permanent_uuid: "58bd123ee5f14926919cf6dd45a975d4" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 40009 } }
I20260929 18:39:55.283293 26995 leader_election.cc:288] T fe433c14506d423eb8e090418d01f129 P 197c331499ce4dfcbcefc644bde81771 [CANDIDATE]: Term 1 election: Requested vote from peers a4b0313e18a04b29a30fde03c6f6be8b (127.13.100.3:34011), 58bd123ee5f14926919cf6dd45a975d4 (127.13.100.1:40009)
I20260929 18:39:55.283777 26642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe433c14506d423eb8e090418d01f129" candidate_uuid: "197c331499ce4dfcbcefc644bde81771" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58bd123ee5f14926919cf6dd45a975d4"
I20260929 18:39:55.283910 26642 raft_consensus.cc:3037] T fe433c14506d423eb8e090418d01f129 P 58bd123ee5f14926919cf6dd45a975d4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:39:55.283979 26986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:55.268814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.2:56227 (local address 127.13.100.3:34011)
0929 18:39:55.269090 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0929 18:39:55.269096 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:39:55.272773 (+ 3677us) server_negotiation.cc:407] Connection header received
0929 18:39:55.272910 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:39:55.272914 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:39:55.272971 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:39:55.273053 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:39:55.277594 (+ 4541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:55.278363 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:55.280202 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:55.280893 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:55.280994 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:55.281047 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0929 18:39:55.283762 (+ 2715us) server_negotiation.cc:299] Negotiation successful
0929 18:39:55.283856 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188}
I20260929 18:39:55.284374 27000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:55.267476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.3:34011 (local address 127.13.100.2:56227)
0929 18:39:55.272673 (+ 5197us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:55.272689 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 18:39:55.272814 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:39:55.273184 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:39:55.273187 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:39:55.273208 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:39:55.273358 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:55.273366 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:55.278619 (+ 5253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:55.278622 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:55.280057 (+ 1435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:55.280068 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:55.283483 (+ 3415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:55.283487 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:55.283607 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:55.283633 (+ 26us) client_negotiation.cc:769] Sending connection context
0929 18:39:55.284158 (+ 525us) client_negotiation.cc:240] Negotiation successful
0929 18:39:55.284239 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5115,"thread_start_us":146,"threads_started":1}
I20260929 18:39:55.285193 26642 raft_consensus.cc:2454] T fe433c14506d423eb8e090418d01f129 P 58bd123ee5f14926919cf6dd45a975d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 197c331499ce4dfcbcefc644bde81771 in term 1.
I20260929 18:39:55.285168 26910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe433c14506d423eb8e090418d01f129" candidate_uuid: "197c331499ce4dfcbcefc644bde81771" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4b0313e18a04b29a30fde03c6f6be8b" is_pre_election: true
I20260929 18:39:55.285470 26910 raft_consensus.cc:2454] T fe433c14506d423eb8e090418d01f129 P a4b0313e18a04b29a30fde03c6f6be8b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197c331499ce4dfcbcefc644bde81771 in term 0.
I20260929 18:39:55.285833 26909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe433c14506d423eb8e090418d01f129" candidate_uuid: "197c331499ce4dfcbcefc644bde81771" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4b0313e18a04b29a30fde03c6f6be8b"
I20260929 18:39:55.285871 26712 leader_election.cc:302] T fe433c14506d423eb8e090418d01f129 P 197c331499ce4dfcbcefc644bde81771 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197c331499ce4dfcbcefc644bde81771, 58bd123ee5f14926919cf6dd45a975d4; no voters:
I20260929 18:39:55.285964 26909 raft_consensus.cc:3037] T fe433c14506d423eb8e090418d01f129 P a4b0313e18a04b29a30fde03c6f6be8b [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:39:55.286098 26995 raft_consensus.cc:2781] T fe433c14506d423eb8e090418d01f129 P 197c331499ce4dfcbcefc644bde81771 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:39:55.286329 26995 raft_consensus.cc:686] T fe433c14506d423eb8e090418d01f129 P 197c331499ce4dfcbcefc644bde81771 [term 1 LEADER]: Becoming Leader. State: Replica: 197c331499ce4dfcbcefc644bde81771, State: Running, Role: LEADER
I20260929 18:39:55.286706 26995 consensus_queue.cc:237] T fe433c14506d423eb8e090418d01f129 P 197c331499ce4dfcbcefc644bde81771 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b0313e18a04b29a30fde03c6f6be8b" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 34011 } } peers { permanent_uuid: "197c331499ce4dfcbcefc644bde81771" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 33813 } } peers { permanent_uuid: "58bd123ee5f14926919cf6dd45a975d4" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 40009 } }
I20260929 18:39:55.287189 26909 raft_consensus.cc:2454] T fe433c14506d423eb8e090418d01f129 P a4b0313e18a04b29a30fde03c6f6be8b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 197c331499ce4dfcbcefc644bde81771 in term 1.
I20260929 18:39:55.290442 26490 catalog_manager.cc:5495] T fe433c14506d423eb8e090418d01f129 P 197c331499ce4dfcbcefc644bde81771 reported cstate change: term changed from 0 to 1, leader changed from <none> to 197c331499ce4dfcbcefc644bde81771 (127.13.100.2). New cstate: current_term: 1 leader_uuid: "197c331499ce4dfcbcefc644bde81771" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b0313e18a04b29a30fde03c6f6be8b" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 34011 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "197c331499ce4dfcbcefc644bde81771" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 33813 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58bd123ee5f14926919cf6dd45a975d4" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 40009 } health_report { overall_health: UNKNOWN } } }
W20260929 18:39:55.300806 26957 tablet.cc:2369] T fe433c14506d423eb8e090418d01f129 P a4b0313e18a04b29a30fde03c6f6be8b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 18:39:55.301945 26824 tablet.cc:2369] T fe433c14506d423eb8e090418d01f129 P 197c331499ce4dfcbcefc644bde81771: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:39:55.309671 26988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:55.304276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53144 (local address 127.13.100.2:33813)
0929 18:39:55.304967 (+ 691us) server_negotiation.cc:206] Beginning negotiation
0929 18:39:55.304972 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:39:55.304987 (+ 15us) server_negotiation.cc:407] Connection header received
0929 18:39:55.305068 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:39:55.305071 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:39:55.305135 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:39:55.305250 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:39:55.305535 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:55.306362 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:55.307568 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:55.307865 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:55.307956 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:55.308347 (+ 391us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:39:55.308894 (+ 547us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:39:55.309016 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0929 18:39:55.309436 (+ 420us) server_negotiation.cc:299] Negotiation successful
0929 18:39:55.309496 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":599}
I20260929 18:39:55.317315 26642 raft_consensus.cc:1261] T fe433c14506d423eb8e090418d01f129 P 58bd123ee5f14926919cf6dd45a975d4 [term 1 FOLLOWER]: Refusing update from remote peer 197c331499ce4dfcbcefc644bde81771: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:39:55.318235 27002 consensus_queue.cc:1035] T fe433c14506d423eb8e090418d01f129 P 197c331499ce4dfcbcefc644bde81771 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58bd123ee5f14926919cf6dd45a975d4" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 40009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:39:55.324230 26909 raft_consensus.cc:1261] T fe433c14506d423eb8e090418d01f129 P a4b0313e18a04b29a30fde03c6f6be8b [term 1 FOLLOWER]: Refusing update from remote peer 197c331499ce4dfcbcefc644bde81771: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:39:55.325076 27002 consensus_queue.cc:1035] T fe433c14506d423eb8e090418d01f129 P 197c331499ce4dfcbcefc644bde81771 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4b0313e18a04b29a30fde03c6f6be8b" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 34011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:39:55.341328 27005 mvcc.cc:205] Tried to move back new op lower bound from 7334736672010002432 to 7334736671897378816. Current Snapshot: MvccSnapshot[applied={T|T < 7334736672010002432 or (T in {7334736672010002432})}]
I20260929 18:39:55.344197 26490 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41970:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:39:55.344372 26490 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41970:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:39:55.347143 26490 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5c93b1d41b784959b6ae9e99ba95b92a: Sending DeleteTablet for 3 replicas of tablet fe433c14506d423eb8e090418d01f129
I20260929 18:39:55.349213 26890 tablet_service.cc:1514] Processing DeleteTablet for tablet fe433c14506d423eb8e090418d01f129 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:39:55 UTC) from {username='kudu', principal='kudu/127.13.100.62@KRBTEST.COM'} at 127.0.0.1:35892
I20260929 18:39:55.350703 26750 tablet_service.cc:1514] Processing DeleteTablet for tablet fe433c14506d423eb8e090418d01f129 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:39:55 UTC) from {username='kudu', principal='kudu/127.13.100.62@KRBTEST.COM'} at 127.0.0.1:53132
I20260929 18:39:55.351166 26622 tablet_service.cc:1514] Processing DeleteTablet for tablet fe433c14506d423eb8e090418d01f129 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:39:55 UTC) from {username='kudu', principal='kudu/127.13.100.62@KRBTEST.COM'} at 127.0.0.1:41762
I20260929 18:39:55.351826 27006 mvcc.cc:205] Tried to move back new op lower bound from 7334736672010002432 to 7334736671897378816. Current Snapshot: MvccSnapshot[applied={T|T < 7334736672010002432}]
I20260929 18:39:55.352468 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 26528
I20260929 18:39:55.354132 27016 tablet_replica.cc:330] T fe433c14506d423eb8e090418d01f129 P 197c331499ce4dfcbcefc644bde81771: stopping tablet replica
I20260929 18:39:55.354455 27016 raft_consensus.cc:2229] T fe433c14506d423eb8e090418d01f129 P 197c331499ce4dfcbcefc644bde81771 [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:39:55.354805 27016 raft_consensus.cc:2258] T fe433c14506d423eb8e090418d01f129 P 197c331499ce4dfcbcefc644bde81771 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:39:55.355183 27015 tablet_replica.cc:330] T fe433c14506d423eb8e090418d01f129 P a4b0313e18a04b29a30fde03c6f6be8b: stopping tablet replica
I20260929 18:39:55.361985 27015 raft_consensus.cc:2229] T fe433c14506d423eb8e090418d01f129 P a4b0313e18a04b29a30fde03c6f6be8b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:39:55.362437 27015 raft_consensus.cc:2258] T fe433c14506d423eb8e090418d01f129 P a4b0313e18a04b29a30fde03c6f6be8b [term 1 FOLLOWER]: Raft consensus is shut down!
W20260929 18:39:55.363255 26477 connection.cc:535] server connection from 127.13.100.1:58831 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260929 18:39:55.363593 26477 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.100.1:40009 (error 108)
I20260929 18:39:55.364296 27015 ts_tablet_manager.cc:1905] T fe433c14506d423eb8e090418d01f129 P a4b0313e18a04b29a30fde03c6f6be8b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:39:55.365531 26987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:55.365148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.1:40009 (local address 127.0.0.1:41776)
0929 18:39:55.365372 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:55.365442 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.1:40009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141}
W20260929 18:39:55.365880 26477 catalog_manager.cc:4584] TS 58bd123ee5f14926919cf6dd45a975d4 (127.13.100.1:40009): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fe433c14506d423eb8e090418d01f129: Network error: Client connection negotiation failed: client connection to 127.13.100.1:40009: connect: Connection refused (error 111)
I20260929 18:39:55.366313 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 26690
I20260929 18:39:55.368674 27015 ts_tablet_manager.cc:1918] T fe433c14506d423eb8e090418d01f129 P a4b0313e18a04b29a30fde03c6f6be8b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:39:55.368773 27015 log.cc:1196] T fe433c14506d423eb8e090418d01f129 P a4b0313e18a04b29a30fde03c6f6be8b: Deleting WAL directory at /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707036041078-13712-0/minicluster-data/ts-2/wal/wals/fe433c14506d423eb8e090418d01f129
I20260929 18:39:55.369125 27015 ts_tablet_manager.cc:1939] T fe433c14506d423eb8e090418d01f129 P a4b0313e18a04b29a30fde03c6f6be8b: Deleting consensus metadata
I20260929 18:39:55.370975 26476 catalog_manager.cc:4841] TS a4b0313e18a04b29a30fde03c6f6be8b (127.13.100.3:34011): tablet fe433c14506d423eb8e090418d01f129 (table test-table [id=abf5cd0007794075a5a2fd994df921ed]) successfully deleted
W20260929 18:39:55.377283 26987 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:39:55.376668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.2:33813 (local address 127.0.0.1:53150)
0929 18:39:55.376860 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:55.376878 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 18:39:55.376971 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:39:55.377173 (+ 202us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.2:33813: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":109}
I20260929 18:39:55.377877 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 26826
I20260929 18:39:55.385516 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 26457
I20260929 18:39:55.394094 13712 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260929 18:39:55.729075 13712 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260929 18:39:55.729188 13712 mini_ranger.cc:66] Stopping Ranger...
I20260929 18:39:56.074564 13712 mini_ranger.cc:68] Stopped Ranger
2026-09-29 18:39:56.074 UTC [25524] LOG: received smart shutdown request
2026-09-29 18:39:56.078 UTC [25524] LOG: background worker "logical replication launcher" (PID 25533) exited with exit code 1
2026-09-29 18:39:56.078 UTC [25528] LOG: shutting down
2026-09-29 18:39:56.246 UTC [25524] LOG: database system is shut down
2026-09-29T18:39:56Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (49191 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707036041078-13712-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:39:56 dist-test-slave-j1xs krb5kdc[27033](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:39:56 dist-test-slave-j1xs krb5kdc[27033](info): set up 2 sockets
Sep 29 18:39:56 dist-test-slave-j1xs krb5kdc[27033](info): commencing operation
krb5kdc: starting...
W20260929 18:39:57.409931 13712 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707036041078-13712-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707036041078-13712-0/krb5kdc/test-user.keytab.
Sep 29 18:39:57 dist-test-slave-j1xs krb5kdc[27033](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707197, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:39:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:39:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.100.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.100.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.100.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.62.keytab.
Entry for principal kudu/127.13.100.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.62.keytab.
I20260929 18:39:57.528846 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707036041078-13712-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707036041078-13712-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707036041078-13712-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.100.62:45105
--webserver_interface=127.13.100.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.62.keytab
--principal=kudu/127.13.100.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.100.20:35979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.100.62:45105,127.13.100.61:45387,127.13.100.60:42013
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707036041078-13712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707036041078-13712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707036041078-13712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:39:57.636561 27046 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:39:57.636884 27046 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:39:57.636971 27046 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:39:57.641026 27046 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:39:57.641119 27046 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:39:57.641157 27046 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:39:57.641211 27046 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:39:57.641250 27046 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:39:57.646538 27046 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.100.20:35979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707036041078-13712-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707036041078-13712-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.100.62:45105,127.13.100.61:45387,127.13.100.60:42013
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.62.keytab
--principal=kudu/127.13.100.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.100.62:45105
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.100.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27046
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707036041078-13712-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:39:57.647611 27046 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:39:57.648631 27046 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:39:57.656962 27054 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:39:57.656976 27052 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:39:57.656965 27051 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:39:57.657357 27046 server_base.cc:1034] running on GCE node
I20260929 18:39:57.657959 27046 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:39:57.659078 27046 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:39:57.660250 27046 hybrid_clock.cc:648] HybridClock initialized: now 1790707197660244 us; error 33 us; skew 500 ppm
Sep 29 18:39:57 dist-test-slave-j1xs krb5kdc[27033](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707197, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:39:57.663062 27046 init.cc:377] Logged in from keytab as kudu/127.13.100.62@KRBTEST.COM (short username kudu)
I20260929 18:39:57.663951 27046 webserver.cc:460] Webserver started at http://127.13.100.62:35293/ using document root <none> and password file <none>
I20260929 18:39:57.664466 27046 fs_manager.cc:359] Metadata directory not provided
I20260929 18:39:57.664526 27046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:39:57.664688 27046 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:39:57.666706 27046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707036041078-13712-0/minicluster-data/master-0/data/instance:
uuid: "024eee637d6a45899a5da0a1fad578de"
format_stamp: "Formatted at 2026-09-29 18:39:57 on dist-test-slave-j1xs"
server_key: "1380a97e7368a240a7b9052830715d8f"
server_key_iv: "ca301c02d71d1f28f57639ef56bb9a61"
server_key_version: "encryptionkey@0"
I20260929 18:39:57.667199 27046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707036041078-13712-0/minicluster-data/master-0/wal/instance:
uuid: "024eee637d6a45899a5da0a1fad578de"
format_stamp: "Formatted at 2026-09-29 18:39:57 on dist-test-slave-j1xs"
server_key: "1380a97e7368a240a7b9052830715d8f"
server_key_iv: "ca301c02d71d1f28f57639ef56bb9a61"
server_key_version: "encryptionkey@0"
I20260929 18:39:57.670874 27046 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 18:39:57.673218 27061 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:39:57.674458 27046 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:39:57.674588 27046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707036041078-13712-0/minicluster-data/master-0/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707036041078-13712-0/minicluster-data/master-0/wal
uuid: "024eee637d6a45899a5da0a1fad578de"
format_stamp: "Formatted at 2026-09-29 18:39:57 on dist-test-slave-j1xs"
server_key: "1380a97e7368a240a7b9052830715d8f"
server_key_iv: "ca301c02d71d1f28f57639ef56bb9a61"
server_key_version: "encryptionkey@0"
I20260929 18:39:57.674695 27046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707036041078-13712-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707036041078-13712-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707036041078-13712-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:39:57.707096 27046 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:39:57.710486 27046 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:39:57.710712 27046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:39:57.718420 27046 rpc_server.cc:307] RPC server started. Bound to: 127.13.100.62:45105
I20260929 18:39:57.718466 27113 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.100.62:45105 every 8 connection(s)
I20260929 18:39:57.719409 27046 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
I20260929 18:39:57.722214 27114 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:39:57.725001 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 27046
I20260929 18:39:57.725147 13712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707036041078-13712-0/minicluster-data/master-0/wal/instance
I20260929 18:39:57.725432 13712 external_mini_cluster.cc:1428] Setting key 39aa83545942886a8d932f021a5b77a5
I20260929 18:39:57.726069 27114 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.100.62" port: 45105 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.13.100.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.100.61@KRBTEST.COM" created.
Sep 29 18:39:57 dist-test-slave-j1xs krb5kdc[27033](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707197, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.62@KRBTEST.COM for kudu/127.13.100.62@KRBTEST.COM
I20260929 18:39:57.741444 27117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:57.728531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:45105 (local address 127.0.0.1:43952)
0929 18:39:57.729154 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:57.729217 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0929 18:39:57.730141 (+ 924us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:39:57.730788 (+ 647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:39:57.730794 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:39:57.731244 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:39:57.731474 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:57.731487 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:57.732605 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:57.732608 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:57.733358 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:57.733367 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:57.733991 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:57.733995 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:57.734109 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:57.734650 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:39:57.734665 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:39:57.736445 (+ 1780us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:39:57.738471 (+ 2026us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:39:57.738479 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:39:57.738487 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:39:57.738742 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:39:57.739055 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:39:57.739058 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:39:57.739060 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:39:57.739169 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:39:57.739735 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:39:57.739741 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:39:57.739870 (+ 129us) client_negotiation.cc:769] Sending connection context
0929 18:39:57.740033 (+ 163us) client_negotiation.cc:240] Negotiation successful
0929 18:39:57.740457 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":398,"mutex_wait_us":155,"thread_start_us":158,"threads_started":1}
I20260929 18:39:57.741444 27118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:57.728528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43952 (local address 127.13.100.62:45105)
0929 18:39:57.729312 (+ 784us) server_negotiation.cc:206] Beginning negotiation
0929 18:39:57.729323 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 18:39:57.729360 (+ 37us) server_negotiation.cc:407] Connection header received
0929 18:39:57.730437 (+ 1077us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:39:57.730441 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:39:57.730613 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:39:57.730793 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:39:57.731616 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:57.732470 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:57.733470 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:57.733823 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:57.733999 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:57.736593 (+ 2594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:39:57.736600 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:39:57.736609 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:39:57.736628 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:39:57.738361 (+ 1733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:39:57.738853 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:39:57.738863 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:39:57.738865 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:39:57.738960 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:39:57.739267 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:39:57.739270 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:39:57.739271 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:39:57.739627 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:39:57.739764 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0929 18:39:57.740110 (+ 346us) server_negotiation.cc:299] Negotiation successful
0929 18:39:57.740408 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":7,"server-negotiator.queue_time_us":656,"spinlock_wait_cycles":231296,"thread_start_us":316,"threads_started":1}
I20260929 18:39:57.744885 27114 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.100.61" port: 45387 } has no permanent_uuid. Determining permanent_uuid...
I20260929 18:39:57.745713 27117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:57.745432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.61:45387 (local address 127.0.0.1:49288)
0929 18:39:57.745571 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:57.745644 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.61:45387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260929 18:39:57.745965 27064 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.100.61:45387: connect: Connection refused (error 111)
I20260929 18:39:57.747144 27117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:57.746908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.61:45387 (local address 127.0.0.1:49294)
0929 18:39:57.747024 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:57.747080 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.61:45387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260929 18:39:57.747321 27114 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.100.61:45387: Network error: Client connection negotiation failed: client connection to 127.13.100.61:45387: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.100.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.61.keytab.
Entry for principal kudu/127.13.100.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.61.keytab.
I20260929 18:39:57.753018 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707036041078-13712-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707036041078-13712-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707036041078-13712-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707036041078-13712-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.100.61:45387
--webserver_interface=127.13.100.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.61.keytab
--principal=kudu/127.13.100.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.100.20:35979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.100.62:45105,127.13.100.61:45387,127.13.100.60:42013
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707036041078-13712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707036041078-13712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707036041078-13712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 18:39:57.796581 27114 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.100.61" port: 45387 } attempt: 1
I20260929 18:39:57.797422 27117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:57.797152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.61:45387 (local address 127.0.0.1:49302)
0929 18:39:57.797286 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:57.797350 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.61:45387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260929 18:39:57.798244 27117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:57.798014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.61:45387 (local address 127.0.0.1:49314)
0929 18:39:57.798145 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:57.798191 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.61:45387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260929 18:39:57.798364 27114 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.100.61:45387: Network error: Client connection negotiation failed: client connection to 127.13.100.61:45387: connect: Connection refused (error 111)
W20260929 18:39:57.864939 27121 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:39:57.865247 27121 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:39:57.865325 27121 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260929 18:39:57.866533 27114 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.100.61" port: 45387 } attempt: 2
I20260929 18:39:57.867308 27117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:57.867019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.61:45387 (local address 127.0.0.1:49316)
0929 18:39:57.867173 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:57.867234 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.61:45387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260929 18:39:57.868037 27117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:57.867805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.61:45387 (local address 127.0.0.1:49320)
0929 18:39:57.867938 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:57.867988 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.61:45387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260929 18:39:57.868180 27114 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.100.61:45387: Network error: Client connection negotiation failed: client connection to 127.13.100.61:45387: connect: Connection refused (error 111)
W20260929 18:39:57.869730 27121 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:39:57.869810 27121 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:39:57.869863 27121 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:39:57.869940 27121 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:39:57.869983 27121 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:39:57.875367 27121 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.100.20:35979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707036041078-13712-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707036041078-13712-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.13.100.62:45105,127.13.100.61:45387,127.13.100.60:42013
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.61.keytab
--principal=kudu/127.13.100.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.100.61:45387
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707036041078-13712-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.13.100.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27121
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707036041078-13712-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:39:57.876708 27121 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:39:57.877748 27121 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:39:57.886070 27129 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:39:57.886140 27121 server_base.cc:1034] running on GCE node
W20260929 18:39:57.886070 27127 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:39:57.886106 27126 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:39:57.886857 27121 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:39:57.887929 27121 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:39:57.889112 27121 hybrid_clock.cc:648] HybridClock initialized: now 1790707197889089 us; error 36 us; skew 500 ppm
Sep 29 18:39:57 dist-test-slave-j1xs krb5kdc[27033](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707197, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:39:57.891899 27121 init.cc:377] Logged in from keytab as kudu/127.13.100.61@KRBTEST.COM (short username kudu)
I20260929 18:39:57.892756 27121 webserver.cc:460] Webserver started at http://127.13.100.61:46437/ using document root <none> and password file <none>
I20260929 18:39:57.893294 27121 fs_manager.cc:359] Metadata directory not provided
I20260929 18:39:57.893380 27121 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:39:57.893599 27121 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:39:57.895632 27121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707036041078-13712-0/minicluster-data/master-1/data/instance:
uuid: "71b8b0c481f744169da626b16e7848f1"
format_stamp: "Formatted at 2026-09-29 18:39:57 on dist-test-slave-j1xs"
server_key: "86468aba86839ac5f6a08e7f3b4a4891"
server_key_iv: "84b769e93d6554d5d3c7faca51a1430a"
server_key_version: "encryptionkey@0"
I20260929 18:39:57.896186 27121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707036041078-13712-0/minicluster-data/master-1/wal/instance:
uuid: "71b8b0c481f744169da626b16e7848f1"
format_stamp: "Formatted at 2026-09-29 18:39:57 on dist-test-slave-j1xs"
server_key: "86468aba86839ac5f6a08e7f3b4a4891"
server_key_iv: "84b769e93d6554d5d3c7faca51a1430a"
server_key_version: "encryptionkey@0"
I20260929 18:39:57.899821 27121 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 18:39:57.902263 27136 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:39:57.903450 27121 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260929 18:39:57.903577 27121 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707036041078-13712-0/minicluster-data/master-1/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707036041078-13712-0/minicluster-data/master-1/wal
uuid: "71b8b0c481f744169da626b16e7848f1"
format_stamp: "Formatted at 2026-09-29 18:39:57 on dist-test-slave-j1xs"
server_key: "86468aba86839ac5f6a08e7f3b4a4891"
server_key_iv: "84b769e93d6554d5d3c7faca51a1430a"
server_key_version: "encryptionkey@0"
I20260929 18:39:57.903692 27121 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707036041078-13712-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707036041078-13712-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707036041078-13712-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:39:57.923492 27121 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:39:57.926762 27121 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:39:57.926963 27121 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:39:57.933619 27121 rpc_server.cc:307] RPC server started. Bound to: 127.13.100.61:45387
I20260929 18:39:57.933665 27188 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.100.61:45387 every 8 connection(s)
I20260929 18:39:57.934576 27121 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707036041078-13712-0/minicluster-data/master-1/data/info.pb
I20260929 18:39:57.937162 27189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:39:57.939616 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 27121
I20260929 18:39:57.939754 13712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707036041078-13712-0/minicluster-data/master-1/wal/instance
I20260929 18:39:57.940030 13712 external_mini_cluster.cc:1428] Setting key ac6ca090aca9b0efdc8aa455116062bb
I20260929 18:39:57.941383 27189 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.100.62" port: 45105 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.13.100.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.100.60@KRBTEST.COM" created.
Sep 29 18:39:57 dist-test-slave-j1xs krb5kdc[27033](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707197, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.61@KRBTEST.COM for kudu/127.13.100.62@KRBTEST.COM
I20260929 18:39:57.954663 27118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:57.943419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43958 (local address 127.13.100.62:45105)
0929 18:39:57.943543 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0929 18:39:57.943547 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:39:57.944529 (+ 982us) server_negotiation.cc:407] Connection header received
0929 18:39:57.945417 (+ 888us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:39:57.945420 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:39:57.945477 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:39:57.945587 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:39:57.946745 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:57.947262 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:57.948384 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:57.948738 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:57.948852 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:57.951294 (+ 2442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:39:57.951297 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:39:57.951299 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:39:57.951322 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:39:57.952682 (+ 1360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:39:57.953168 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:39:57.953173 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:39:57.953175 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:39:57.953258 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:39:57.953574 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:39:57.953579 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:39:57.953581 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:39:57.953759 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:39:57.953862 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 18:39:57.954368 (+ 506us) server_negotiation.cc:299] Negotiation successful
0929 18:39:57.954492 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 18:39:57.955399 27192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:57.943861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:45105 (local address 127.0.0.1:43958)
0929 18:39:57.944381 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:57.944421 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 18:39:57.945222 (+ 801us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:39:57.945741 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:39:57.945748 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:39:57.946158 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:39:57.946608 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:57.946620 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:57.947382 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:57.947391 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:57.948264 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:57.948272 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:57.948886 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:57.948891 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:57.949056 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:57.949578 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:39:57.949595 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:39:57.951113 (+ 1518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:39:57.952796 (+ 1683us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:39:57.952802 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:39:57.952814 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:39:57.953070 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:39:57.953346 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:39:57.953350 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:39:57.953352 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:39:57.953484 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:39:57.953859 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:39:57.953865 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:39:57.954127 (+ 262us) client_negotiation.cc:769] Sending connection context
0929 18:39:57.954301 (+ 174us) client_negotiation.cc:240] Negotiation successful
0929 18:39:57.954602 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":143,"threads_started":1}
I20260929 18:39:57.956687 27189 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.100.61" port: 45387 } has no permanent_uuid. Determining permanent_uuid...
I20260929 18:39:57.959329 27114 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.100.61" port: 45387 } attempt: 3
Sep 29 18:39:57 dist-test-slave-j1xs krb5kdc[27033](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707197, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.61@KRBTEST.COM for kudu/127.13.100.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.100.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.60.keytab.
Entry for principal kudu/127.13.100.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.60.keytab.
Sep 29 18:39:57 dist-test-slave-j1xs krb5kdc[27033](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707197, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.62@KRBTEST.COM for kudu/127.13.100.61@KRBTEST.COM
I20260929 18:39:57.965387 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707036041078-13712-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707036041078-13712-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707036041078-13712-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707036041078-13712-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.100.60:42013
--webserver_interface=127.13.100.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.60.keytab
--principal=kudu/127.13.100.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.100.20:35979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.100.62:45105,127.13.100.61:45387,127.13.100.60:42013
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707036041078-13712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707036041078-13712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707036041078-13712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 18:39:57.967353 27192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:57.957208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.61:45387 (local address 127.0.0.1:49334)
0929 18:39:57.957310 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:57.957321 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 18:39:57.957379 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:39:57.958039 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:39:57.958042 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:39:57.958172 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:39:57.958268 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:57.958273 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:57.959327 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:57.959331 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:57.960388 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:57.960397 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:57.961146 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:57.961149 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:57.961234 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:57.961765 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:39:57.961776 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:39:57.963157 (+ 1381us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:39:57.965441 (+ 2284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:39:57.965443 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:39:57.965445 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:39:57.965686 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:39:57.966052 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:39:57.966058 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:39:57.966062 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:39:57.966143 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:39:57.966689 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:39:57.966695 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:39:57.966824 (+ 129us) client_negotiation.cc:769] Sending connection context
0929 18:39:57.966958 (+ 134us) client_negotiation.cc:240] Negotiation successful
0929 18:39:57.967106 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260929 18:39:57.967367 27194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:57.957352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49334 (local address 127.13.100.61:45387)
0929 18:39:57.957629 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0929 18:39:57.957637 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 18:39:57.957653 (+ 16us) server_negotiation.cc:407] Connection header received
0929 18:39:57.957682 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:39:57.957686 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:39:57.957822 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:39:57.958047 (+ 225us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:39:57.958365 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:57.959198 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:57.960494 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:57.961011 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:57.961116 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:57.963302 (+ 2186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:39:57.963314 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:39:57.963331 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:39:57.963356 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:39:57.965326 (+ 1970us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:39:57.965805 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:39:57.965814 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:39:57.965818 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:39:57.965955 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:39:57.966251 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:39:57.966257 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:39:57.966262 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:39:57.966568 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:39:57.966698 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0929 18:39:57.967106 (+ 408us) server_negotiation.cc:299] Negotiation successful
0929 18:39:57.967237 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":121,"threads_started":1}
I20260929 18:39:57.969301 27189 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.100.60" port: 42013 } has no permanent_uuid. Determining permanent_uuid...
I20260929 18:39:57.970194 27192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:57.969790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.60:42013 (local address 127.0.0.1:51672)
0929 18:39:57.970039 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:57.970118 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.60:42013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162}
W20260929 18:39:57.970484 27140 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.100.60:42013: connect: Connection refused (error 111)
I20260929 18:39:57.971084 27117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:57.959703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.61:45387 (local address 127.0.0.1:49350)
0929 18:39:57.959859 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:57.959874 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 18:39:57.959952 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:39:57.960268 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:39:57.960271 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:39:57.960440 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:39:57.960560 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:57.960566 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:57.961538 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:57.961543 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:57.962354 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:57.962370 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:57.962989 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:57.962993 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:57.963128 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:57.963603 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:39:57.963615 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:39:57.964917 (+ 1302us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:39:57.969390 (+ 4473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:39:57.969394 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:39:57.969396 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:39:57.969722 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:39:57.970038 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:39:57.970044 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:39:57.970049 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:39:57.970162 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:39:57.970573 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:39:57.970576 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:39:57.970678 (+ 102us) client_negotiation.cc:769] Sending connection context
0929 18:39:57.970780 (+ 102us) client_negotiation.cc:240] Negotiation successful
0929 18:39:57.970921 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89}
I20260929 18:39:57.971107 27195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:57.959819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49350 (local address 127.13.100.61:45387)
0929 18:39:57.960088 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0929 18:39:57.960095 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 18:39:57.960105 (+ 10us) server_negotiation.cc:407] Connection header received
0929 18:39:57.960131 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:39:57.960133 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:39:57.960162 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:39:57.960234 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:39:57.960650 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:57.961371 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:57.962515 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:57.962826 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:57.962974 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:57.965079 (+ 2105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:39:57.965083 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:39:57.965086 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:39:57.965115 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:39:57.969175 (+ 4060us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:39:57.969818 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:39:57.969821 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:39:57.969822 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:39:57.969941 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:39:57.970290 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:39:57.970294 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:39:57.970295 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:39:57.970478 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:39:57.970581 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 18:39:57.970843 (+ 262us) server_negotiation.cc:299] Negotiation successful
0929 18:39:57.970949 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":213,"thread_start_us":111,"threads_started":1}
I20260929 18:39:57.971834 27114 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.100.60" port: 42013 } has no permanent_uuid. Determining permanent_uuid...
I20260929 18:39:57.971943 27192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:57.971581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.60:42013 (local address 127.0.0.1:51684)
0929 18:39:57.971794 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:57.971859 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.60:42013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134}
W20260929 18:39:57.972076 27189 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.100.60:42013: Network error: Client connection negotiation failed: client connection to 127.13.100.60:42013: connect: Connection refused (error 111)
I20260929 18:39:57.972590 27117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:57.972273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.60:42013 (local address 127.0.0.1:51698)
0929 18:39:57.972419 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:57.972489 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.60:42013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260929 18:39:57.973397 27117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:57.973059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.60:42013 (local address 127.0.0.1:51704)
0929 18:39:57.973241 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:57.973308 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.60:42013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
W20260929 18:39:57.973575 27114 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.100.60:42013: Network error: Client connection negotiation failed: client connection to 127.13.100.60:42013: connect: Connection refused (error 111)
I20260929 18:39:58.004766 27114 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.100.60" port: 42013 } attempt: 1
I20260929 18:39:58.005672 27117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:58.005367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.60:42013 (local address 127.0.0.1:51706)
0929 18:39:58.005534 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:58.005596 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.60:42013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260929 18:39:58.006420 27117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:58.006211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.60:42013 (local address 127.0.0.1:51718)
0929 18:39:58.006302 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:58.006365 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.60:42013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260929 18:39:58.006626 27114 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.100.60:42013: Network error: Client connection negotiation failed: client connection to 127.13.100.60:42013: connect: Connection refused (error 111)
I20260929 18:39:58.021322 27189 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.100.60" port: 42013 } attempt: 1
I20260929 18:39:58.022230 27192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:58.021874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.60:42013 (local address 127.0.0.1:51720)
0929 18:39:58.022080 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:58.022150 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.60:42013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260929 18:39:58.023002 27192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:58.022776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.60:42013 (local address 127.0.0.1:51732)
0929 18:39:58.022896 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:58.022949 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.60:42013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260929 18:39:58.023128 27189 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.100.60:42013: Network error: Client connection negotiation failed: client connection to 127.13.100.60:42013: connect: Connection refused (error 111)
W20260929 18:39:58.080240 27196 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:39:58.080525 27196 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:39:58.080608 27196 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260929 18:39:58.081795 27114 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.100.60" port: 42013 } attempt: 2
I20260929 18:39:58.082598 27117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:58.082305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.60:42013 (local address 127.0.0.1:51744)
0929 18:39:58.082467 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:58.082526 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.60:42013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260929 18:39:58.083261 27117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:58.083044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.60:42013 (local address 127.0.0.1:51756)
0929 18:39:58.083165 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:58.083212 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.60:42013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260929 18:39:58.083380 27114 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.100.60:42013: Network error: Client connection negotiation failed: client connection to 127.13.100.60:42013: connect: Connection refused (error 111)
W20260929 18:39:58.085050 27196 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:39:58.085124 27196 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:39:58.085168 27196 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:39:58.085199 27196 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:39:58.085238 27196 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:39:58.090855 27196 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.100.20:35979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707036041078-13712-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707036041078-13712-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.13.100.62:45105,127.13.100.61:45387,127.13.100.60:42013
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.60.keytab
--principal=kudu/127.13.100.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.100.60:42013
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707036041078-13712-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.13.100.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27196
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707036041078-13712-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:39:58.091271 27189 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.100.60" port: 42013 } attempt: 2
I20260929 18:39:58.091936 27196 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:39:58.091960 27192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:58.091672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.60:42013 (local address 127.0.0.1:51758)
0929 18:39:58.091798 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:58.091855 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.60:42013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260929 18:39:58.092645 27192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:58.092468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.60:42013 (local address 127.0.0.1:51766)
0929 18:39:58.092559 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:58.092600 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.60:42013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260929 18:39:58.092761 27189 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.100.60:42013: Network error: Client connection negotiation failed: client connection to 127.13.100.60:42013: connect: Connection refused (error 111)
I20260929 18:39:58.093153 27196 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:39:58.101380 27203 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:39:58.101393 27205 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:39:58.101380 27202 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:39:58.101706 27196 server_base.cc:1034] running on GCE node
I20260929 18:39:58.102363 27196 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:39:58.103443 27196 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:39:58.104600 27196 hybrid_clock.cc:648] HybridClock initialized: now 1790707198104565 us; error 48 us; skew 500 ppm
Sep 29 18:39:58 dist-test-slave-j1xs krb5kdc[27033](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707198, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:39:58.107409 27196 init.cc:377] Logged in from keytab as kudu/127.13.100.60@KRBTEST.COM (short username kudu)
I20260929 18:39:58.108480 27196 webserver.cc:460] Webserver started at http://127.13.100.60:42835/ using document root <none> and password file <none>
I20260929 18:39:58.109037 27196 fs_manager.cc:359] Metadata directory not provided
I20260929 18:39:58.109125 27196 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:39:58.109341 27196 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:39:58.111387 27196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707036041078-13712-0/minicluster-data/master-2/data/instance:
uuid: "9d1bf1382b4940e6bee91b1383dd0d59"
format_stamp: "Formatted at 2026-09-29 18:39:58 on dist-test-slave-j1xs"
server_key: "aa1269f0daaaba7c966be78e8380e0f9"
server_key_iv: "bf84cb6ac29df9c99bc5a989ead9bc7a"
server_key_version: "encryptionkey@0"
I20260929 18:39:58.111955 27196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707036041078-13712-0/minicluster-data/master-2/wal/instance:
uuid: "9d1bf1382b4940e6bee91b1383dd0d59"
format_stamp: "Formatted at 2026-09-29 18:39:58 on dist-test-slave-j1xs"
server_key: "aa1269f0daaaba7c966be78e8380e0f9"
server_key_iv: "bf84cb6ac29df9c99bc5a989ead9bc7a"
server_key_version: "encryptionkey@0"
I20260929 18:39:58.115682 27196 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 18:39:58.118152 27212 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:39:58.119386 27196 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:39:58.119535 27196 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707036041078-13712-0/minicluster-data/master-2/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707036041078-13712-0/minicluster-data/master-2/wal
uuid: "9d1bf1382b4940e6bee91b1383dd0d59"
format_stamp: "Formatted at 2026-09-29 18:39:58 on dist-test-slave-j1xs"
server_key: "aa1269f0daaaba7c966be78e8380e0f9"
server_key_iv: "bf84cb6ac29df9c99bc5a989ead9bc7a"
server_key_version: "encryptionkey@0"
I20260929 18:39:58.119659 27196 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707036041078-13712-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707036041078-13712-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707036041078-13712-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:39:58.151029 27196 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:39:58.154120 27196 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:39:58.154333 27196 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:39:58.161415 27196 rpc_server.cc:307] RPC server started. Bound to: 127.13.100.60:42013
I20260929 18:39:58.161438 27264 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.100.60:42013 every 8 connection(s)
I20260929 18:39:58.162479 27196 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707036041078-13712-0/minicluster-data/master-2/data/info.pb
I20260929 18:39:58.163139 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 27196
I20260929 18:39:58.163280 13712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707036041078-13712-0/minicluster-data/master-2/wal/instance
I20260929 18:39:58.163565 13712 external_mini_cluster.cc:1428] Setting key 803843daf0809056bc41cda4a9aacad3
I20260929 18:39:58.165688 27265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:39:58.170109 27265 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.100.62" port: 45105 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 18:39:58 dist-test-slave-j1xs krb5kdc[27033](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707197, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.100.60@KRBTEST.COM
Sep 29 18:39:58 dist-test-slave-j1xs krb5kdc[27033](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707198, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.60@KRBTEST.COM for kudu/127.13.100.62@KRBTEST.COM
I20260929 18:39:58.178373 27268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:58.164940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51770 (local address 127.13.100.60:42013)
0929 18:39:58.165387 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0929 18:39:58.165401 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 18:39:58.165436 (+ 35us) server_negotiation.cc:407] Connection header received
0929 18:39:58.166165 (+ 729us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:39:58.166183 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:39:58.166509 (+ 326us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:39:58.166836 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:39:58.167641 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:58.168618 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:58.169445 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:58.170007 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:58.170169 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:58.172875 (+ 2706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:39:58.172882 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:39:58.172894 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:39:58.172924 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:39:58.174963 (+ 2039us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:39:58.175493 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:39:58.175499 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:39:58.175503 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:39:58.175628 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:39:58.175965 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:39:58.175970 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:39:58.175972 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:39:58.176536 (+ 564us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:39:58.176719 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0929 18:39:58.177030 (+ 311us) server_negotiation.cc:299] Negotiation successful
0929 18:39:58.177351 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":153,"threads_started":1}
I20260929 18:39:58.181488 27269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:58.171786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:45105 (local address 127.0.0.1:43966)
0929 18:39:58.172232 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:58.172288 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0929 18:39:58.172410 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:39:58.172751 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:39:58.172759 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:39:58.173252 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:39:58.173407 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:58.173415 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:58.174196 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:58.174200 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:58.175146 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:58.175153 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:58.175734 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:58.175741 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:58.175851 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:58.176344 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:39:58.176358 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:39:58.177957 (+ 1599us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:39:58.179807 (+ 1850us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:39:58.179813 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:39:58.179817 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:39:58.180087 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:39:58.180368 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:39:58.180372 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:39:58.180374 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:39:58.180455 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:39:58.180896 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:39:58.180901 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:39:58.181022 (+ 121us) client_negotiation.cc:769] Sending connection context
0929 18:39:58.181170 (+ 148us) client_negotiation.cc:240] Negotiation successful
0929 18:39:58.181346 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":178,"threads_started":1}
I20260929 18:39:58.181577 27118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:58.171791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43966 (local address 127.13.100.62:45105)
0929 18:39:58.171975 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0929 18:39:58.171981 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:39:58.172366 (+ 385us) server_negotiation.cc:407] Connection header received
0929 18:39:58.172513 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:39:58.172518 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:39:58.172571 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:39:58.172700 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:39:58.173539 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:58.174084 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:58.175256 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:58.175602 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:58.175697 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:58.178148 (+ 2451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:39:58.178151 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:39:58.178153 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:39:58.178181 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:39:58.179710 (+ 1529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:39:58.180189 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:39:58.180195 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:39:58.180197 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:39:58.180273 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:39:58.180539 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:39:58.180545 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:39:58.180549 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:39:58.180799 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:39:58.180892 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 18:39:58.181254 (+ 362us) server_negotiation.cc:299] Negotiation successful
0929 18:39:58.181377 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260929 18:39:58.182497 27265 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.100.61" port: 45387 } has no permanent_uuid. Determining permanent_uuid...
I20260929 18:39:58.182528 27114 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.100.60" port: 42013 } attempt: 3
I20260929 18:39:58.183913 27189 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.100.60" port: 42013 } attempt: 3
Sep 29 18:39:58 dist-test-slave-j1xs krb5kdc[27033](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707197, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.100.62@KRBTEST.COM
Sep 29 18:39:58 dist-test-slave-j1xs krb5kdc[27033](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707198, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.60@KRBTEST.COM for kudu/127.13.100.61@KRBTEST.COM
Sep 29 18:39:58 dist-test-slave-j1xs krb5kdc[27033](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707197, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.62@KRBTEST.COM for kudu/127.13.100.60@KRBTEST.COM
Sep 29 18:39:58 dist-test-slave-j1xs krb5kdc[27033](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707197, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.61@KRBTEST.COM for kudu/127.13.100.60@KRBTEST.COM
I20260929 18:39:58.191903 27270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:58.180631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43974 (local address 127.13.100.62:45105)
0929 18:39:58.180921 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0929 18:39:58.180938 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 18:39:58.180950 (+ 12us) server_negotiation.cc:407] Connection header received
0929 18:39:58.180982 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:39:58.180985 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:39:58.181025 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:39:58.181129 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:39:58.181850 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:58.182602 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:58.183666 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:58.184013 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:58.184533 (+ 520us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:58.187103 (+ 2570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:39:58.187106 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:39:58.187111 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:39:58.187140 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:39:58.189689 (+ 2549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:39:58.190264 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:39:58.190269 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:39:58.190274 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:39:58.190380 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:39:58.190750 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:39:58.190755 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:39:58.190759 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:39:58.190998 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:39:58.191197 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0929 18:39:58.191545 (+ 348us) server_negotiation.cc:299] Negotiation successful
0929 18:39:58.191691 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225,"thread_start_us":113,"threads_started":1}
I20260929 18:39:58.193802 27195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:58.183065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49358 (local address 127.13.100.61:45387)
0929 18:39:58.183220 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0929 18:39:58.183224 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:39:58.183433 (+ 209us) server_negotiation.cc:407] Connection header received
0929 18:39:58.183592 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:39:58.183595 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:39:58.183668 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:39:58.183720 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:39:58.184351 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:58.185082 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:58.186288 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:58.186662 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:58.186733 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:58.189572 (+ 2839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:39:58.189579 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:39:58.189585 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:39:58.189613 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:39:58.191366 (+ 1753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:39:58.191918 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:39:58.191923 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:39:58.191925 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:39:58.192007 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:39:58.192739 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:39:58.192743 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:39:58.192747 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:39:58.193032 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:39:58.193114 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 18:39:58.193483 (+ 369us) server_negotiation.cc:299] Negotiation successful
0929 18:39:58.193619 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260929 18:39:58.194074 27117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:58.182872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.60:42013 (local address 127.0.0.1:51776)
0929 18:39:58.183074 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:58.183086 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:39:58.183153 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:39:58.183527 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:39:58.183530 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:39:58.183694 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:39:58.183766 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:58.183771 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:58.185138 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:58.185145 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:58.186107 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:58.186118 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:58.186754 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:58.186760 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:58.186850 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:58.187700 (+ 850us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:39:58.187718 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:39:58.189468 (+ 1750us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:39:58.192451 (+ 2983us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:39:58.192456 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:39:58.192461 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:39:58.192710 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:39:58.193022 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:39:58.193025 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:39:58.193026 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:39:58.193114 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:39:58.193568 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:39:58.193571 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:39:58.193663 (+ 92us) client_negotiation.cc:769] Sending connection context
0929 18:39:58.193744 (+ 81us) client_negotiation.cc:240] Negotiation successful
0929 18:39:58.193915 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140}
I20260929 18:39:58.194664 27269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:58.182979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.61:45387 (local address 127.0.0.1:49358)
0929 18:39:58.183371 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:58.183383 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:39:58.183491 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:39:58.183835 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:39:58.183837 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:39:58.184093 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:39:58.184223 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:58.184230 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:58.185204 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:58.185207 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:58.186119 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:58.186132 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:58.186914 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:58.186918 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:58.187014 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:58.187531 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:39:58.187553 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:39:58.189404 (+ 1851us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:39:58.191473 (+ 2069us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:39:58.191479 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:39:58.191484 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:39:58.191801 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:39:58.192468 (+ 667us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:39:58.192471 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:39:58.192473 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:39:58.192595 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:39:58.193188 (+ 593us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:39:58.193191 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:39:58.193292 (+ 101us) client_negotiation.cc:769] Sending connection context
0929 18:39:58.194342 (+ 1050us) client_negotiation.cc:240] Negotiation successful
0929 18:39:58.194486 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338}
I20260929 18:39:58.194895 27192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:58.184296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.60:42013 (local address 127.0.0.1:51792)
0929 18:39:58.184511 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:58.184523 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:39:58.184592 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:39:58.185195 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:39:58.185198 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:39:58.185448 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:39:58.185550 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:58.185557 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:58.187076 (+ 1519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:58.187082 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:58.187860 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:58.187870 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:58.188534 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:58.188540 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:58.188652 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:58.189098 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:39:58.189115 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:39:58.190540 (+ 1425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:39:58.193293 (+ 2753us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:39:58.193295 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:39:58.193297 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:39:58.193534 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:39:58.193827 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:39:58.193831 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:39:58.193833 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:39:58.193957 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:39:58.194433 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:39:58.194437 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:39:58.194526 (+ 89us) client_negotiation.cc:769] Sending connection context
0929 18:39:58.194606 (+ 80us) client_negotiation.cc:240] Negotiation successful
0929 18:39:58.194733 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148}
I20260929 18:39:58.195439 27271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:58.184709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51792 (local address 127.13.100.60:42013)
0929 18:39:58.184967 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0929 18:39:58.184974 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 18:39:58.184985 (+ 11us) server_negotiation.cc:407] Connection header received
0929 18:39:58.185014 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:39:58.185016 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:39:58.185050 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:39:58.185670 (+ 620us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:39:58.185761 (+ 91us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:58.186948 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:58.188017 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:58.188416 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:58.188518 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:58.190680 (+ 2162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:39:58.190683 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:39:58.190686 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:39:58.190712 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:39:58.193186 (+ 2474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:39:58.193651 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:39:58.193654 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:39:58.193655 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:39:58.193728 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:39:58.194096 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:39:58.194099 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:39:58.194101 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:39:58.194303 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:39:58.195170 (+ 867us) server_negotiation.cc:1035] Waiting for connection context
0929 18:39:58.195240 (+ 70us) server_negotiation.cc:299] Negotiation successful
0929 18:39:58.195332 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":89,"threads_started":1}
I20260929 18:39:58.195912 27265 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.100.60" port: 42013 } has no permanent_uuid. Determining permanent_uuid...
I20260929 18:39:58.194809 27268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:58.183271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51776 (local address 127.13.100.60:42013)
0929 18:39:58.183359 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0929 18:39:58.183362 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:39:58.183371 (+ 9us) server_negotiation.cc:407] Connection header received
0929 18:39:58.183397 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:39:58.183399 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:39:58.183429 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:39:58.183512 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:39:58.183884 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:58.184976 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:58.186260 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:58.186620 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:58.187883 (+ 1263us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:58.189631 (+ 1748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:39:58.189635 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:39:58.189638 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:39:58.189661 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:39:58.192329 (+ 2668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:39:58.192838 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:39:58.192841 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:39:58.192842 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:39:58.192920 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:39:58.193247 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:39:58.193250 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:39:58.193252 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:39:58.193453 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:39:58.194424 (+ 971us) server_negotiation.cc:1035] Waiting for connection context
0929 18:39:58.194506 (+ 82us) server_negotiation.cc:299] Negotiation successful
0929 18:39:58.194632 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260929 18:39:58.199268 27189 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71b8b0c481f744169da626b16e7848f1: Bootstrap starting.
Sep 29 18:39:58 dist-test-slave-j1xs krb5kdc[27033](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707197, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.100.61@KRBTEST.COM
I20260929 18:39:58.201558 27114 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 024eee637d6a45899a5da0a1fad578de: Bootstrap starting.
I20260929 18:39:58.202150 27189 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 71b8b0c481f744169da626b16e7848f1: Neither blocks nor log segments found. Creating new log.
Sep 29 18:39:58 dist-test-slave-j1xs krb5kdc[27033](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707198, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.60@KRBTEST.COM for kudu/127.13.100.60@KRBTEST.COM
I20260929 18:39:58.203073 27189 log.cc:824] T 00000000000000000000000000000000 P 71b8b0c481f744169da626b16e7848f1: Log is configured to *not* fsync() on all Append() calls
I20260929 18:39:58.204512 27194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:58.193509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49364 (local address 127.13.100.61:45387)
0929 18:39:58.193625 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0929 18:39:58.193629 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:39:58.193942 (+ 313us) server_negotiation.cc:407] Connection header received
0929 18:39:58.193981 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:39:58.193984 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:39:58.194027 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:39:58.194075 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:39:58.194938 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:58.195703 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:58.196788 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:58.197099 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:58.197188 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:58.200292 (+ 3104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:39:58.200295 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:39:58.200298 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:39:58.200324 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:39:58.202231 (+ 1907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:39:58.202753 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:39:58.202757 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:39:58.202759 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:39:58.202859 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:39:58.203213 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:39:58.203217 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:39:58.203219 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:39:58.203455 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:39:58.203639 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0929 18:39:58.204123 (+ 484us) server_negotiation.cc:299] Negotiation successful
0929 18:39:58.204359 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 18:39:58.204681 27114 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 024eee637d6a45899a5da0a1fad578de: Neither blocks nor log segments found. Creating new log.
I20260929 18:39:58.205804 27114 log.cc:824] T 00000000000000000000000000000000 P 024eee637d6a45899a5da0a1fad578de: Log is configured to *not* fsync() on all Append() calls
I20260929 18:39:58.206305 27269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:58.196485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.60:42013 (local address 127.0.0.1:51806)
0929 18:39:58.196631 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0929 18:39:58.196643 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:39:58.196716 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:39:58.196905 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:39:58.196907 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:39:58.197063 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:39:58.197126 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:58.197130 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:58.199043 (+ 1913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:58.199047 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:58.199753 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:39:58.199764 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:39:58.200375 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:39:58.200378 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:39:58.200450 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:58.201068 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:39:58.201081 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:39:58.202581 (+ 1500us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:39:58.204846 (+ 2265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:39:58.204851 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:39:58.204855 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:39:58.205062 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:39:58.205337 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:39:58.205340 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:39:58.205341 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:39:58.205414 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:39:58.205815 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:39:58.205817 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:39:58.205908 (+ 91us) client_negotiation.cc:769] Sending connection context
0929 18:39:58.206010 (+ 102us) client_negotiation.cc:240] Negotiation successful
0929 18:39:58.206132 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91}
I20260929 18:39:58.206730 27271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:58.196508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51806 (local address 127.13.100.60:42013)
0929 18:39:58.196595 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0929 18:39:58.196599 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:39:58.196681 (+ 82us) server_negotiation.cc:407] Connection header received
0929 18:39:58.196781 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:39:58.196786 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:39:58.196818 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:39:58.196896 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:39:58.197225 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:58.197803 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:58.199882 (+ 2079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:58.200187 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:58.200278 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:58.202714 (+ 2436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:39:58.202718 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:39:58.202720 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:39:58.202742 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:39:58.204739 (+ 1997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:39:58.205169 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:39:58.205172 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:39:58.205173 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:39:58.205247 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:39:58.205507 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:39:58.205509 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:39:58.205511 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:39:58.205698 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:39:58.205784 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 18:39:58.206492 (+ 708us) server_negotiation.cc:299] Negotiation successful
0929 18:39:58.206597 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260929 18:39:58.207916 27189 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71b8b0c481f744169da626b16e7848f1: No bootstrap required, opened a new log
I20260929 18:39:58.207929 27114 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 024eee637d6a45899a5da0a1fad578de: No bootstrap required, opened a new log
I20260929 18:39:58.208138 13712 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:39:58.211993 27114 raft_consensus.cc:348] T 00000000000000000000000000000000 P 024eee637d6a45899a5da0a1fad578de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "024eee637d6a45899a5da0a1fad578de" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 45105 } } peers { permanent_uuid: "71b8b0c481f744169da626b16e7848f1" member_type: VOTER last_known_addr { host: "127.13.100.61" port: 45387 } } peers { permanent_uuid: "9d1bf1382b4940e6bee91b1383dd0d59" member_type: VOTER last_known_addr { host: "127.13.100.60" port: 42013 } }
I20260929 18:39:58.212216 27114 raft_consensus.cc:374] T 00000000000000000000000000000000 P 024eee637d6a45899a5da0a1fad578de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:39:58.212276 27114 raft_consensus.cc:729] T 00000000000000000000000000000000 P 024eee637d6a45899a5da0a1fad578de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 024eee637d6a45899a5da0a1fad578de, State: Initialized, Role: FOLLOWER
I20260929 18:39:58.212707 27114 consensus_queue.cc:260] T 00000000000000000000000000000000 P 024eee637d6a45899a5da0a1fad578de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "024eee637d6a45899a5da0a1fad578de" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 45105 } } peers { permanent_uuid: "71b8b0c481f744169da626b16e7848f1" member_type: VOTER last_known_addr { host: "127.13.100.61" port: 45387 } } peers { permanent_uuid: "9d1bf1382b4940e6bee91b1383dd0d59" member_type: VOTER last_known_addr { host: "127.13.100.60" port: 42013 } }
I20260929 18:39:58.214056 27114 sys_catalog.cc:564] T 00000000000000000000000000000000 P 024eee637d6a45899a5da0a1fad578de [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:39:58.215133 27281 sys_catalog.cc:455] T 00000000000000000000000000000000 P 024eee637d6a45899a5da0a1fad578de [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "024eee637d6a45899a5da0a1fad578de" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 45105 } } peers { permanent_uuid: "71b8b0c481f744169da626b16e7848f1" member_type: VOTER last_known_addr { host: "127.13.100.61" port: 45387 } } peers { permanent_uuid: "9d1bf1382b4940e6bee91b1383dd0d59" member_type: VOTER last_known_addr { host: "127.13.100.60" port: 42013 } } }
I20260929 18:39:58.215723 27189 raft_consensus.cc:348] T 00000000000000000000000000000000 P 71b8b0c481f744169da626b16e7848f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "024eee637d6a45899a5da0a1fad578de" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 45105 } } peers { permanent_uuid: "71b8b0c481f744169da626b16e7848f1" member_type: VOTER last_known_addr { host: "127.13.100.61" port: 45387 } } peers { permanent_uuid: "9d1bf1382b4940e6bee91b1383dd0d59" member_type: VOTER last_known_addr { host: "127.13.100.60" port: 42013 } }
I20260929 18:39:58.215981 27189 raft_consensus.cc:374] T 00000000000000000000000000000000 P 71b8b0c481f744169da626b16e7848f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:39:58.216039 27189 raft_consensus.cc:729] T 00000000000000000000000000000000 P 71b8b0c481f744169da626b16e7848f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71b8b0c481f744169da626b16e7848f1, State: Initialized, Role: FOLLOWER
I20260929 18:39:58.216856 27281 sys_catalog.cc:458] T 00000000000000000000000000000000 P 024eee637d6a45899a5da0a1fad578de [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:39:58.219880 27265 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d1bf1382b4940e6bee91b1383dd0d59: Bootstrap starting.
W20260929 18:39:58.220677 27292 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 024eee637d6a45899a5da0a1fad578de: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:39:58.220813 27292 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:39:58.222886 27265 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9d1bf1382b4940e6bee91b1383dd0d59: Neither blocks nor log segments found. Creating new log.
I20260929 18:39:58.219926 27046 master_runner.cc:186] Error getting master registration for 127.13.100.61:45387: OK, instance_id { permanent_uuid: "71b8b0c481f744169da626b16e7848f1" instance_seqno: 1790707197931815 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 18:39:58.223773 27046 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 18:39:58.224023 27189 consensus_queue.cc:260] T 00000000000000000000000000000000 P 71b8b0c481f744169da626b16e7848f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "024eee637d6a45899a5da0a1fad578de" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 45105 } } peers { permanent_uuid: "71b8b0c481f744169da626b16e7848f1" member_type: VOTER last_known_addr { host: "127.13.100.61" port: 45387 } } peers { permanent_uuid: "9d1bf1382b4940e6bee91b1383dd0d59" member_type: VOTER last_known_addr { host: "127.13.100.60" port: 42013 } }
I20260929 18:39:58.224911 27270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:58.210020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43978 (local address 127.13.100.62:45105)
0929 18:39:58.210424 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0929 18:39:58.210431 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 18:39:58.210637 (+ 206us) server_negotiation.cc:407] Connection header received
0929 18:39:58.210829 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:39:58.210833 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:39:58.210874 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:39:58.210921 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:39:58.211951 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:58.212680 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:58.214008 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:58.214332 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:58.214425 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:58.216875 (+ 2450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:39:58.216880 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:39:58.216883 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:39:58.216909 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:39:58.219179 (+ 2270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:39:58.219689 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:39:58.219692 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:39:58.219694 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:39:58.219779 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:39:58.220073 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:39:58.220076 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:39:58.220078 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:39:58.220320 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:39:58.220444 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0929 18:39:58.224616 (+ 4172us) server_negotiation.cc:299] Negotiation successful
0929 18:39:58.224752 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330}
I20260929 18:39:58.226226 27194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:58.210019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49376 (local address 127.13.100.61:45387)
0929 18:39:58.210429 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0929 18:39:58.210433 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:39:58.210750 (+ 317us) server_negotiation.cc:407] Connection header received
0929 18:39:58.210912 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:39:58.210915 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:39:58.210957 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:39:58.211003 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:39:58.211956 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:58.212663 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:58.213873 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:58.214346 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:58.214427 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:58.216354 (+ 1927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:39:58.216357 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:39:58.216360 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:39:58.216385 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:39:58.223764 (+ 7379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:39:58.224375 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:39:58.224382 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:39:58.224387 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:39:58.224476 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:39:58.225003 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:39:58.225007 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:39:58.225009 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:39:58.225260 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:39:58.225391 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0929 18:39:58.225914 (+ 523us) server_negotiation.cc:299] Negotiation successful
0929 18:39:58.226043 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336}
I20260929 18:39:58.227916 27189 sys_catalog.cc:564] T 00000000000000000000000000000000 P 71b8b0c481f744169da626b16e7848f1 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:39:58.228506 27265 log.cc:824] T 00000000000000000000000000000000 P 9d1bf1382b4940e6bee91b1383dd0d59: Log is configured to *not* fsync() on all Append() calls
I20260929 18:39:58.230266 27271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:39:58.216335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51818 (local address 127.13.100.60:42013)
0929 18:39:58.216608 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0929 18:39:58.216613 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:39:58.221160 (+ 4547us) server_negotiation.cc:407] Connection header received
0929 18:39:58.221203 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:39:58.221207 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:39:58.221266 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:39:58.221331 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:39:58.222104 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:58.222899 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:58.224028 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:39:58.224354 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:39:58.224457 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:39:58.226621 (+ 2164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:39:58.226625 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:39:58.226628 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:39:58.226655 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:39:58.228337 (+ 1682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:39:58.228826 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:39:58.228829 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:39:58.228831 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:39:58.228917 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:39:58.229236 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:39:58.229239 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:39:58.229241 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:39:58.229450 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:39:58.229594 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0929 18:39:58.229880 (+ 286us) server_negotiation.cc:299] Negotiation successful
0929 18:39:58.230070 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200}
I20260929 18:39:58.230965 27294 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71b8b0c481f744169da626b16e7848f1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "024eee637d6a45899a5da0a1fad578de" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 45105 } } peers { permanent_uuid: "71b8b0c481f744169da626b16e7848f1" member_type: VOTER last_known_addr { host: "127.13.100.61" port: 45387 } } peers { permanent_uuid: "9d1bf1382b4940e6bee91b1383dd0d59" member_type: VOTER last_known_addr { host: "127.13.100.60" port: 42013 } } }
I20260929 18:39:58.231598 27294 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71b8b0c481f744169da626b16e7848f1 [sys.catalog]: This master's current role is: FOLLOWER
W20260929 18:39:58.234047 27306 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 71b8b0c481f744169da626b16e7848f1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:39:58.234127 27306 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:39:58.235251 27121 master_runner.cc:186] Error getting master registration for 127.13.100.60:42013: OK, instance_id { permanent_uuid: "9d1bf1382b4940e6bee91b1383dd0d59" instance_seqno: 1790707198159360 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 18:39:58.238540 27121 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 18:39:58.244050 27265 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d1bf1382b4940e6bee91b1383dd0d59: No bootstrap required, opened a new log
I20260929 18:39:58.248265 27265 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9d1bf1382b4940e6bee91b1383dd0d59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "024eee637d6a45899a5da0a1fad578de" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 45105 } } peers { permanent_uuid: "71b8b0c481f744169da626b16e7848f1" member_type: VOTER last_known_addr { host: "127.13.100.61" port: 45387 } } peers { permanent_uuid: "9d1bf1382b4940e6bee91b1383dd0d59" member_type: VOTER last_known_addr { host: "127.13.100.60" port: 42013 } }
I20260929 18:39:58.248513 27265 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9d1bf1382b4940e6bee91b1383dd0d59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:39:58.248588 27265 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9d1bf1382b4940e6bee91b1383dd0d59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d1bf1382b4940e6bee91b1383dd0d59, State: Initialized, Role: FOLLOWER
I20260929 18:39:58.249048 27265 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9d1bf1382b4940e6bee91b1383dd0d59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "024eee637d6a45899a5da0a1fad578de" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 45105 } } peers { permanent_uuid: "71b8b0c481f744169da626b16e7848f1" member_type: VOTER last_known_addr { host: "127.13.100.61" port: 45387 } } peers { permanent_uuid: "9d1bf1382b4940e6bee91b1383dd0d59" member_type: VOTER last_known_addr { host: "127.13.100.60" port: 42013 } }
I20260929 18:39:58.249527 27308 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d1bf1382b4940e6bee91b1383dd0d59 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "024eee637d6a45899a5da0a1fad578de" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 45105 } } peers { permanent_uuid: "71b8b0c481f744169da626b16e7848f1" member_type: VOTER last_known_addr { host: "127.13.100.61" port: 45387 } } peers { permanent_uuid: "9d1bf1382b4940e6bee91b1383dd0d59" member_type: VOTER last_known_addr { host: "127.13.100.60" port: 42013 } } }
I20260929 18:39:58.251097 27308 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d1bf1382b4940e6bee91b1383dd0d59 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:39:58.250443 27265 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9d1bf1382b4940e6bee91b1383dd0d59 [sys.catalog]: configured and running, proceeding with master startup.
W20260929 18:39:58.256621 27319 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9d1bf1382b4940e6bee91b1383dd0d59: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:39:58.256758 27319 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:39:58.259668 27196 master_runner.cc:263] No leader master found from master 9d1bf1382b4940e6bee91b1383dd0d59
I20260929 18:39:58.259733 27196 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 18:39:58.388331 27308 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9d1bf1382b4940e6bee91b1383dd0d59 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:39:58.388556 27308 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9d1bf1382b4940e6bee91b1383dd0d59 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "024eee637d6a45899a5da0a1fad578de" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 45105 } } peers { permanent_uuid: "71b8b0c481f744169da626b16e7848f1" member_type: VOTER last_known_addr { host: "127.13.100.61" port: 45387 } } peers { permanent_uuid: "9d1bf1382b4940e6bee91b1383dd0d59" member_type: VOTER last_known_addr { host: "127.13.100.60" port: 42013 } }
I20260929 18:39:58.389602 27308 leader_election.cc:288] T 00000000000000000000000000000000 P 9d1bf1382b4940e6bee91b1383dd0d59 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 024eee637d6a45899a5da0a1fad578de (127.13.100.62:45105), 71b8b0c481f744169da626b16e7848f1 (127.13.100.61:45387)
I20260929 18:39:58.390197 27086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9d1bf1382b4940e6bee91b1383dd0d59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "024eee637d6a45899a5da0a1fad578de" is_pre_election: true
I20260929 18:39:58.390509 27086 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 024eee637d6a45899a5da0a1fad578de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d1bf1382b4940e6bee91b1383dd0d59 in term 0.
I20260929 18:39:58.390921 27213 leader_election.cc:302] T 00000000000000000000000000000000 P 9d1bf1382b4940e6bee91b1383dd0d59 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 024eee637d6a45899a5da0a1fad578de, 9d1bf1382b4940e6bee91b1383dd0d59; no voters:
I20260929 18:39:58.391229 27308 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9d1bf1382b4940e6bee91b1383dd0d59 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:39:58.391324 27308 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9d1bf1382b4940e6bee91b1383dd0d59 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:39:58.391373 27308 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9d1bf1382b4940e6bee91b1383dd0d59 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:39:58.391423 27163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9d1bf1382b4940e6bee91b1383dd0d59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71b8b0c481f744169da626b16e7848f1" is_pre_election: true
I20260929 18:39:58.391695 27163 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 71b8b0c481f744169da626b16e7848f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d1bf1382b4940e6bee91b1383dd0d59 in term 0.
I20260929 18:39:58.392563 27308 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9d1bf1382b4940e6bee91b1383dd0d59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "024eee637d6a45899a5da0a1fad578de" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 45105 } } peers { permanent_uuid: "71b8b0c481f744169da626b16e7848f1" member_type: VOTER last_known_addr { host: "127.13.100.61" port: 45387 } } peers { permanent_uuid: "9d1bf1382b4940e6bee91b1383dd0d59" member_type: VOTER last_known_addr { host: "127.13.100.60" port: 42013 } }
I20260929 18:39:58.392925 27308 leader_election.cc:288] T 00000000000000000000000000000000 P 9d1bf1382b4940e6bee91b1383dd0d59 [CANDIDATE]: Term 1 election: Requested vote from peers 024eee637d6a45899a5da0a1fad578de (127.13.100.62:45105), 71b8b0c481f744169da626b16e7848f1 (127.13.100.61:45387)
I20260929 18:39:58.393262 27086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9d1bf1382b4940e6bee91b1383dd0d59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "024eee637d6a45899a5da0a1fad578de"
I20260929 18:39:58.393287 27163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9d1bf1382b4940e6bee91b1383dd0d59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71b8b0c481f744169da626b16e7848f1"
I20260929 18:39:58.393373 27086 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 024eee637d6a45899a5da0a1fad578de [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:39:58.393386 27163 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 71b8b0c481f744169da626b16e7848f1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:39:58.394599 27086 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 024eee637d6a45899a5da0a1fad578de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d1bf1382b4940e6bee91b1383dd0d59 in term 1.
I20260929 18:39:58.394713 27163 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 71b8b0c481f744169da626b16e7848f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d1bf1382b4940e6bee91b1383dd0d59 in term 1.
I20260929 18:39:58.394925 27213 leader_election.cc:302] T 00000000000000000000000000000000 P 9d1bf1382b4940e6bee91b1383dd0d59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 024eee637d6a45899a5da0a1fad578de, 9d1bf1382b4940e6bee91b1383dd0d59; no voters:
I20260929 18:39:58.395092 27308 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9d1bf1382b4940e6bee91b1383dd0d59 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:39:58.395355 27308 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9d1bf1382b4940e6bee91b1383dd0d59 [term 1 LEADER]: Becoming Leader. State: Replica: 9d1bf1382b4940e6bee91b1383dd0d59, State: Running, Role: LEADER
I20260929 18:39:58.395658 27308 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9d1bf1382b4940e6bee91b1383dd0d59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "024eee637d6a45899a5da0a1fad578de" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 45105 } } peers { permanent_uuid: "71b8b0c481f744169da626b16e7848f1" member_type: VOTER last_known_addr { host: "127.13.100.61" port: 45387 } } peers { permanent_uuid: "9d1bf1382b4940e6bee91b1383dd0d59" member_type: VOTER last_known_addr { host: "127.13.100.60" port: 42013 } }
I20260929 18:39:58.397447 27321 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d1bf1382b4940e6bee91b1383dd0d59 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9d1bf1382b4940e6bee91b1383dd0d59. Latest consensus state: current_term: 1 leader_uuid: "9d1bf1382b4940e6bee91b1383dd0d59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "024eee637d6a45899a5da0a1fad578de" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 45105 } } peers { permanent_uuid: "71b8b0c481f744169da626b16e7848f1" member_type: VOTER last_known_addr { host: "127.13.100.61" port: 45387 } } peers { permanent_uuid: "9d1bf1382b4940e6bee91b1383dd0d59" member_type: VOTER last_known_addr { host: "127.13.100.60" port: 42013 } } }
I20260929 18:39:58.397562 27321 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d1bf1382b4940e6bee91b1383dd0d59 [sys.catalog]: This master's current role is: LEADER
I20260929 18:39:58.397974 27323 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:39:58.398934 27323 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:39:58.403318 27163 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 71b8b0c481f744169da626b16e7848f1 [term 1 FOLLOWER]: Refusing update from remote peer 9d1bf1382b4940e6bee91b1383dd0d59: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:39:58.403349 27086 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 024eee637d6a45899a5da0a1fad578de [term 1 FOLLOWER]: Refusing update from remote peer 9d1bf1382b4940e6bee91b1383dd0d59: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:39:58.403949 27308 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9d1bf1382b4940e6bee91b1383dd0d59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71b8b0c481f744169da626b16e7848f1" member_type: VOTER last_known_addr { host: "127.13.100.61" port: 45387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:39:58.404255 27321 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9d1bf1382b4940e6bee91b1383dd0d59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "024eee637d6a45899a5da0a1fad578de" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 45105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 18:39:58.407202 27218 tablet.cc:2369] T 00000000000000000000000000000000 P 9d1bf1382b4940e6bee91b1383dd0d59: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:39:58.408532 27281 sys_catalog.cc:455] T 00000000000000000000000000000000 P 024eee637d6a45899a5da0a1fad578de [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9d1bf1382b4940e6bee91b1383dd0d59. Latest consensus state: current_term: 1 leader_uuid: "9d1bf1382b4940e6bee91b1383dd0d59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "024eee637d6a45899a5da0a1fad578de" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 45105 } } peers { permanent_uuid: "71b8b0c481f744169da626b16e7848f1" member_type: VOTER last_known_addr { host: "127.13.100.61" port: 45387 } } peers { permanent_uuid: "9d1bf1382b4940e6bee91b1383dd0d59" member_type: VOTER last_known_addr { host: "127.13.100.60" port: 42013 } } }
I20260929 18:39:58.408679 27281 sys_catalog.cc:458] T 00000000000000000000000000000000 P 024eee637d6a45899a5da0a1fad578de [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:39:58.410734 27294 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71b8b0c481f744169da626b16e7848f1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9d1bf1382b4940e6bee91b1383dd0d59. Latest consensus state: current_term: 1 leader_uuid: "9d1bf1382b4940e6bee91b1383dd0d59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "024eee637d6a45899a5da0a1fad578de" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 45105 } } peers { permanent_uuid: "71b8b0c481f744169da626b16e7848f1" member_type: VOTER last_known_addr { host: "127.13.100.61" port: 45387 } } peers { permanent_uuid: "9d1bf1382b4940e6bee91b1383dd0d59" member_type: VOTER last_known_addr { host: "127.13.100.60" port: 42013 } } }
I20260929 18:39:58.410846 27294 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71b8b0c481f744169da626b16e7848f1 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:39:58.413357 27325 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d1bf1382b4940e6bee91b1383dd0d59 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9d1bf1382b4940e6bee91b1383dd0d59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "024eee637d6a45899a5da0a1fad578de" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 45105 } } peers { permanent_uuid: "71b8b0c481f744169da626b16e7848f1" member_type: VOTER last_known_addr { host: "127.13.100.61" port: 45387 } } peers { permanent_uuid: "9d1bf1382b4940e6bee91b1383dd0d59" member_type: VOTER last_known_addr { host: "127.13.100.60" port: 42013 } } }
I20260929 18:39:58.413508 27325 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d1bf1382b4940e6bee91b1383dd0d59 [sys.catalog]: This master's current role is: LEADER
I20260929 18:39:58.413594 27324 mvcc.cc:205] Tried to move back new op lower bound from 7334736684652670976 to 7334736684631203840. Current Snapshot: MvccSnapshot[applied={T|T < 7334736684652670976}]
I20260929 18:39:58.414103 27308 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d1bf1382b4940e6bee91b1383dd0d59 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9d1bf1382b4940e6bee91b1383dd0d59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "024eee637d6a45899a5da0a1fad578de" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 45105 } } peers { permanent_uuid: "71b8b0c481f744169da626b16e7848f1" member_type: VOTER last_known_addr { host: "127.13.100.61" port: 45387 } } peers { permanent_uuid: "9d1bf1382b4940e6bee91b1383dd0d59" member_type: VOTER last_known_addr { host: "127.13.100.60" port: 42013 } } }
I20260929 18:39:58.414230 27308 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d1bf1382b4940e6bee91b1383dd0d59 [sys.catalog]: This master's current role is: LEADER
I20260929 18:39:58.415134 27281 sys_catalog.cc:455] T 00000000000000000000000000000000 P 024eee637d6a45899a5da0a1fad578de [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9d1bf1382b4940e6bee91b1383dd0d59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "024eee637d6a45899a5da0a1fad578de" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 45105 } } peers { permanent_uuid: "71b8b0c481f744169da626b16e7848f1" member_type: VOTER last_known_addr { host: "127.13.100.61" port: 45387 } } peers { permanent_uuid: "9d1bf1382b4940e6bee91b1383dd0d59" member_type: VOTER last_known_addr { host: "127.13.100.60" port: 42013 } } }
I20260929 18:39:58.415153 27294 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71b8b0c481f744169da626b16e7848f1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9d1bf1382b4940e6bee91b1383dd0d59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "024eee637d6a45899a5da0a1fad578de" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 45105 } } peers { permanent_uuid: "71b8b0c481f744169da626b16e7848f1" member_type: VOTER last_known_addr { host: "127.13.100.61" port: 45387 } } peers { permanent_uuid: "9d1bf1382b4940e6bee91b1383dd0d59" member_type: VOTER last_known_addr { host: "127.13.100.60" port: 42013 } } }
I20260929 18:39:58.415252 27281 sys_catalog.cc:458] T 00000000000000000000000000000000 P 024eee637d6a45899a5da0a1fad578de [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:39:58.415262 27294 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71b8b0c481f744169da626b16e7848f1 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:39:58.415656 27327 mvcc.cc:205] Tried to move back new op lower bound from 7334736684652670976 to 7334736684631203840. Current Snapshot: MvccSnapshot[applied={T|T < 7334736684652670976}]
I20260929 18:39:58.415673 27326 mvcc.cc:205] Tried to move back new op lower bound from 7334736684652670976 to 7334736684631203840. Current Snapshot: MvccSnapshot[applied={T|T < 7334736684652670976}]
I20260929 18:39:58.418653 27323 catalog_manager.cc:1347] Generated new cluster ID: 8f9430a3064d4456bb2e7a13bc34f419
I20260929 18:39:58.418708 27323 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:39:58.453905 27323 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:39:58.455338 27323 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:39:58.463860 27323 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9d1bf1382b4940e6bee91b1383dd0d59: Generated new TSK 0
I20260929 18:39:58.464529 27323 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:39:59.222270 27292 catalog_manager.cc:1259] Loaded cluster ID: 8f9430a3064d4456bb2e7a13bc34f419
I20260929 18:39:59.222355 27292 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 024eee637d6a45899a5da0a1fad578de: loading cluster ID for follower catalog manager: success
I20260929 18:39:59.224059 27292 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 024eee637d6a45899a5da0a1fad578de: acquiring CA information for follower catalog manager: success
I20260929 18:39:59.225176 27292 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 024eee637d6a45899a5da0a1fad578de: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 18:39:59.235272 27306 catalog_manager.cc:1259] Loaded cluster ID: 8f9430a3064d4456bb2e7a13bc34f419
I20260929 18:39:59.235351 27306 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 71b8b0c481f744169da626b16e7848f1: loading cluster ID for follower catalog manager: success
I20260929 18:39:59.236871 27306 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 71b8b0c481f744169da626b16e7848f1: acquiring CA information for follower catalog manager: success
I20260929 18:39:59.237941 27306 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 71b8b0c481f744169da626b16e7848f1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 18:39:59.522785 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 27046
I20260929 18:39:59.529328 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 27121
I20260929 18:39:59.535542 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 27196
2026-09-29T18:39:59Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3176 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790707036041078-13712-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:39:59 dist-test-slave-j1xs krb5kdc[27339](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:39:59 dist-test-slave-j1xs krb5kdc[27339](info): set up 2 sockets
Sep 29 18:39:59 dist-test-slave-j1xs krb5kdc[27339](info): commencing operation
krb5kdc: starting...
W20260929 18:40:00.590101 13712 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790707036041078-13712-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790707036041078-13712-0/krb5kdc/test-user.keytab.
Sep 29 18:40:00 dist-test-slave-j1xs krb5kdc[27339](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707200, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:40:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:40:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.100.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.100.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.100.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.62.keytab.
Entry for principal kudu/127.13.100.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.62.keytab.
I20260929 18:40:00.709285 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790707036041078-13712-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790707036041078-13712-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790707036041078-13712-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.100.62:42675
--webserver_interface=127.13.100.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.62.keytab
--principal=kudu/127.13.100.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.100.20:41257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.100.62:42675
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790707036041078-13712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790707036041078-13712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790707036041078-13712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:40:00.817641 27352 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:40:00.817984 27352 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:40:00.818070 27352 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:40:00.822082 27352 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:40:00.822168 27352 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:40:00.822208 27352 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:40:00.822264 27352 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:40:00.822300 27352 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:40:00.827549 27352 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.100.20:41257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790707036041078-13712-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790707036041078-13712-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.100.62:42675
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.62.keytab
--principal=kudu/127.13.100.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.100.62:42675
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.100.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27352
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790707036041078-13712-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:40:00.828650 27352 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:40:00.829675 27352 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:40:00.838236 27360 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:40:00.838202 27357 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:40:00.838312 27352 server_base.cc:1034] running on GCE node
W20260929 18:40:00.838220 27358 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:40:00.839071 27352 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:40:00.840094 27352 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:40:00.841274 27352 hybrid_clock.cc:648] HybridClock initialized: now 1790707200841263 us; error 29 us; skew 500 ppm
Sep 29 18:40:00 dist-test-slave-j1xs krb5kdc[27339](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707200, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:40:00.848145 27352 init.cc:377] Logged in from keytab as kudu/127.13.100.62@KRBTEST.COM (short username kudu)
I20260929 18:40:00.849040 27352 webserver.cc:460] Webserver started at http://127.13.100.62:39399/ using document root <none> and password file <none>
I20260929 18:40:00.849622 27352 fs_manager.cc:359] Metadata directory not provided
I20260929 18:40:00.849706 27352 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:40:00.849979 27352 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:40:00.851991 27352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790707036041078-13712-0/minicluster-data/master-0/data/instance:
uuid: "59c43ec4d17147afbabbcce31accf5b6"
format_stamp: "Formatted at 2026-09-29 18:40:00 on dist-test-slave-j1xs"
server_key: "a7cb98ed9002c99a2b3425e081bddb6b"
server_key_iv: "38e834c4a4a06582fbb30a569f8cad06"
server_key_version: "encryptionkey@0"
I20260929 18:40:00.852545 27352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790707036041078-13712-0/minicluster-data/master-0/wal/instance:
uuid: "59c43ec4d17147afbabbcce31accf5b6"
format_stamp: "Formatted at 2026-09-29 18:40:00 on dist-test-slave-j1xs"
server_key: "a7cb98ed9002c99a2b3425e081bddb6b"
server_key_iv: "38e834c4a4a06582fbb30a569f8cad06"
server_key_version: "encryptionkey@0"
I20260929 18:40:00.856365 27352 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 18:40:00.858803 27367 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:40:00.860018 27352 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:40:00.860150 27352 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790707036041078-13712-0/minicluster-data/master-0/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790707036041078-13712-0/minicluster-data/master-0/wal
uuid: "59c43ec4d17147afbabbcce31accf5b6"
format_stamp: "Formatted at 2026-09-29 18:40:00 on dist-test-slave-j1xs"
server_key: "a7cb98ed9002c99a2b3425e081bddb6b"
server_key_iv: "38e834c4a4a06582fbb30a569f8cad06"
server_key_version: "encryptionkey@0"
I20260929 18:40:00.860280 27352 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790707036041078-13712-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790707036041078-13712-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790707036041078-13712-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:40:00.874703 27352 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:40:00.878006 27352 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:40:00.878196 27352 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:40:00.885545 27352 rpc_server.cc:307] RPC server started. Bound to: 127.13.100.62:42675
I20260929 18:40:00.885572 27419 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.100.62:42675 every 8 connection(s)
I20260929 18:40:00.886518 27352 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
I20260929 18:40:00.889194 27420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:40:00.894986 27420 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59c43ec4d17147afbabbcce31accf5b6: Bootstrap starting.
I20260929 18:40:00.895432 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 27352
I20260929 18:40:00.895565 13712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790707036041078-13712-0/minicluster-data/master-0/wal/instance
I20260929 18:40:00.895874 13712 external_mini_cluster.cc:1428] Setting key 8de1b2c7ba28e3b0011e0fcaab97f141
I20260929 18:40:00.897823 27420 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 59c43ec4d17147afbabbcce31accf5b6: Neither blocks nor log segments found. Creating new log.
I20260929 18:40:00.898571 27420 log.cc:824] T 00000000000000000000000000000000 P 59c43ec4d17147afbabbcce31accf5b6: Log is configured to *not* fsync() on all Append() calls
I20260929 18:40:00.900408 27420 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59c43ec4d17147afbabbcce31accf5b6: No bootstrap required, opened a new log
I20260929 18:40:00.903863 27420 raft_consensus.cc:348] T 00000000000000000000000000000000 P 59c43ec4d17147afbabbcce31accf5b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59c43ec4d17147afbabbcce31accf5b6" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 42675 } }
I20260929 18:40:00.904136 27420 raft_consensus.cc:374] T 00000000000000000000000000000000 P 59c43ec4d17147afbabbcce31accf5b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:40:00.904220 27420 raft_consensus.cc:729] T 00000000000000000000000000000000 P 59c43ec4d17147afbabbcce31accf5b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59c43ec4d17147afbabbcce31accf5b6, State: Initialized, Role: FOLLOWER
Sep 29 18:40:00 dist-test-slave-j1xs krb5kdc[27339](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707200, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.100.62@KRBTEST.COM
I20260929 18:40:00.904750 27420 consensus_queue.cc:260] T 00000000000000000000000000000000 P 59c43ec4d17147afbabbcce31accf5b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59c43ec4d17147afbabbcce31accf5b6" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 42675 } }
I20260929 18:40:00.904943 27420 raft_consensus.cc:388] T 00000000000000000000000000000000 P 59c43ec4d17147afbabbcce31accf5b6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:40:00.905018 27420 raft_consensus.cc:482] T 00000000000000000000000000000000 P 59c43ec4d17147afbabbcce31accf5b6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:40:00.905149 27420 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 59c43ec4d17147afbabbcce31accf5b6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:40:00.906184 27420 raft_consensus.cc:504] T 00000000000000000000000000000000 P 59c43ec4d17147afbabbcce31accf5b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59c43ec4d17147afbabbcce31accf5b6" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 42675 } }
I20260929 18:40:00.906695 27420 leader_election.cc:302] T 00000000000000000000000000000000 P 59c43ec4d17147afbabbcce31accf5b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 59c43ec4d17147afbabbcce31accf5b6; no voters:
I20260929 18:40:00.907104 27420 leader_election.cc:288] T 00000000000000000000000000000000 P 59c43ec4d17147afbabbcce31accf5b6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:40:00.907186 27425 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 59c43ec4d17147afbabbcce31accf5b6 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:40:00.907379 27425 raft_consensus.cc:686] T 00000000000000000000000000000000 P 59c43ec4d17147afbabbcce31accf5b6 [term 1 LEADER]: Becoming Leader. State: Replica: 59c43ec4d17147afbabbcce31accf5b6, State: Running, Role: LEADER
I20260929 18:40:00.907749 27425 consensus_queue.cc:237] T 00000000000000000000000000000000 P 59c43ec4d17147afbabbcce31accf5b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59c43ec4d17147afbabbcce31accf5b6" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 42675 } }
I20260929 18:40:00.908450 27420 sys_catalog.cc:564] T 00000000000000000000000000000000 P 59c43ec4d17147afbabbcce31accf5b6 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:40:00.908953 27425 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59c43ec4d17147afbabbcce31accf5b6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 59c43ec4d17147afbabbcce31accf5b6. Latest consensus state: current_term: 1 leader_uuid: "59c43ec4d17147afbabbcce31accf5b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59c43ec4d17147afbabbcce31accf5b6" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 42675 } } }
I20260929 18:40:00.909072 27425 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59c43ec4d17147afbabbcce31accf5b6 [sys.catalog]: This master's current role is: LEADER
I20260929 18:40:00.910126 27426 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59c43ec4d17147afbabbcce31accf5b6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "59c43ec4d17147afbabbcce31accf5b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59c43ec4d17147afbabbcce31accf5b6" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 42675 } } }
I20260929 18:40:00.910275 27426 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59c43ec4d17147afbabbcce31accf5b6 [sys.catalog]: This master's current role is: LEADER
I20260929 18:40:00.911074 27423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:00.897288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38450 (local address 127.13.100.62:42675)
0929 18:40:00.897739 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0929 18:40:00.897753 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 18:40:00.897789 (+ 36us) server_negotiation.cc:407] Connection header received
0929 18:40:00.898503 (+ 714us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:40:00.898520 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:40:00.898881 (+ 361us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:40:00.899209 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:40:00.900058 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:00.901080 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:00.901993 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:00.902465 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:00.902627 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:40:00.905506 (+ 2879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:40:00.905512 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:40:00.905526 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:40:00.905555 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:40:00.907686 (+ 2131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:40:00.908211 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:40:00.908216 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:40:00.908220 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:40:00.908326 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:40:00.908619 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:40:00.908623 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:40:00.908625 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:40:00.909293 (+ 668us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:40:00.909493 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0929 18:40:00.909763 (+ 270us) server_negotiation.cc:299] Negotiation successful
0929 18:40:00.910110 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":134,"threads_started":1}
W20260929 18:40:00.913347 27440 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 59c43ec4d17147afbabbcce31accf5b6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:40:00.913426 27440 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:40:00.913487 27430 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:40:00.914216 13712 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:40:00.914409 27430 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:40:00.920035 27430 catalog_manager.cc:1347] Generated new cluster ID: 3a9964cdfb8b4533bc4e80e8c5e67785
I20260929 18:40:00.920109 27430 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:40:00.934285 27430 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:40:00.935597 27430 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:40:00.943526 27430 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 59c43ec4d17147afbabbcce31accf5b6: Generated new TSK 0
I20260929 18:40:00.944387 27430 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:40:00.970854 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 27352
2026-09-29T18:40:00Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1433 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:40:01 dist-test-slave-j1xs krb5kdc[27447](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:40:01 dist-test-slave-j1xs krb5kdc[27447](info): set up 2 sockets
Sep 29 18:40:01 dist-test-slave-j1xs krb5kdc[27447](info): commencing operation
krb5kdc: starting...
W20260929 18:40:02.018183 13712 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/krb5kdc/test-user.keytab.
Sep 29 18:40:02 dist-test-slave-j1xs krb5kdc[27447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707202, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:40:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:40:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.100.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.100.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.100.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.62.keytab.
Entry for principal kudu/127.13.100.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.62.keytab.
I20260929 18:40:02.140894 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.100.62:35827
--webserver_interface=127.13.100.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.62.keytab
--principal=kudu/127.13.100.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.100.20:43917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.100.62:35827
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:40:02.250888 27460 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:40:02.251142 27460 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:40:02.251181 27460 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:40:02.255105 27460 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:40:02.255155 27460 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:40:02.255174 27460 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:40:02.255191 27460 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:40:02.255205 27460 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:40:02.260262 27460 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.100.20:43917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.100.62:35827
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.62.keytab
--principal=kudu/127.13.100.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.100.62:35827
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.100.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27460
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:40:02.261241 27460 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:40:02.262226 27460 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:40:02.270298 27466 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:40:02.270416 27460 server_base.cc:1034] running on GCE node
W20260929 18:40:02.270298 27465 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:40:02.270393 27468 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:40:02.271103 27460 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:40:02.272178 27460 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:40:02.273365 27460 hybrid_clock.cc:648] HybridClock initialized: now 1790707202273343 us; error 34 us; skew 500 ppm
Sep 29 18:40:02 dist-test-slave-j1xs krb5kdc[27447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707202, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:40:02.276141 27460 init.cc:377] Logged in from keytab as kudu/127.13.100.62@KRBTEST.COM (short username kudu)
I20260929 18:40:02.277019 27460 webserver.cc:460] Webserver started at http://127.13.100.62:46467/ using document root <none> and password file <none>
I20260929 18:40:02.277567 27460 fs_manager.cc:359] Metadata directory not provided
I20260929 18:40:02.277655 27460 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:40:02.277882 27460 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:40:02.279917 27460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/master-0/data/instance:
uuid: "bc5af9cdcf5d4b16b729c8ac3ceb361c"
format_stamp: "Formatted at 2026-09-29 18:40:02 on dist-test-slave-j1xs"
server_key: "f78f34ab993a73e964650e60e2595883"
server_key_iv: "29027fd7bcfe9d91d74d334f1f49ab83"
server_key_version: "encryptionkey@0"
I20260929 18:40:02.280493 27460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/master-0/wal/instance:
uuid: "bc5af9cdcf5d4b16b729c8ac3ceb361c"
format_stamp: "Formatted at 2026-09-29 18:40:02 on dist-test-slave-j1xs"
server_key: "f78f34ab993a73e964650e60e2595883"
server_key_iv: "29027fd7bcfe9d91d74d334f1f49ab83"
server_key_version: "encryptionkey@0"
I20260929 18:40:02.284170 27460 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 18:40:02.286826 27475 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:40:02.288017 27460 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 18:40:02.288162 27460 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/master-0/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/master-0/wal
uuid: "bc5af9cdcf5d4b16b729c8ac3ceb361c"
format_stamp: "Formatted at 2026-09-29 18:40:02 on dist-test-slave-j1xs"
server_key: "f78f34ab993a73e964650e60e2595883"
server_key_iv: "29027fd7bcfe9d91d74d334f1f49ab83"
server_key_version: "encryptionkey@0"
I20260929 18:40:02.288267 27460 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:40:02.308452 27460 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:40:02.311573 27460 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:40:02.311803 27460 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:40:02.318575 27460 rpc_server.cc:307] RPC server started. Bound to: 127.13.100.62:35827
I20260929 18:40:02.318854 27527 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.100.62:35827 every 8 connection(s)
I20260929 18:40:02.319537 27460 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
I20260929 18:40:02.322193 27528 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:40:02.327059 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 27460
I20260929 18:40:02.327193 13712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/master-0/wal/instance
I20260929 18:40:02.327489 13712 external_mini_cluster.cc:1428] Setting key dda51e81b31059c34e4f244ac87372a9
I20260929 18:40:02.327620 27528 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc5af9cdcf5d4b16b729c8ac3ceb361c: Bootstrap starting.
I20260929 18:40:02.330039 27528 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bc5af9cdcf5d4b16b729c8ac3ceb361c: Neither blocks nor log segments found. Creating new log.
I20260929 18:40:02.330883 27528 log.cc:824] T 00000000000000000000000000000000 P bc5af9cdcf5d4b16b729c8ac3ceb361c: Log is configured to *not* fsync() on all Append() calls
I20260929 18:40:02.332827 27528 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc5af9cdcf5d4b16b729c8ac3ceb361c: No bootstrap required, opened a new log
Sep 29 18:40:02 dist-test-slave-j1xs krb5kdc[27447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707202, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.100.62@KRBTEST.COM
I20260929 18:40:02.337184 27528 raft_consensus.cc:348] T 00000000000000000000000000000000 P bc5af9cdcf5d4b16b729c8ac3ceb361c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc5af9cdcf5d4b16b729c8ac3ceb361c" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 35827 } }
I20260929 18:40:02.337390 27528 raft_consensus.cc:374] T 00000000000000000000000000000000 P bc5af9cdcf5d4b16b729c8ac3ceb361c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:40:02.337468 27528 raft_consensus.cc:729] T 00000000000000000000000000000000 P bc5af9cdcf5d4b16b729c8ac3ceb361c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc5af9cdcf5d4b16b729c8ac3ceb361c, State: Initialized, Role: FOLLOWER
I20260929 18:40:02.337988 27528 consensus_queue.cc:260] T 00000000000000000000000000000000 P bc5af9cdcf5d4b16b729c8ac3ceb361c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc5af9cdcf5d4b16b729c8ac3ceb361c" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 35827 } }
I20260929 18:40:02.338126 27528 raft_consensus.cc:388] T 00000000000000000000000000000000 P bc5af9cdcf5d4b16b729c8ac3ceb361c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:40:02.338193 27528 raft_consensus.cc:482] T 00000000000000000000000000000000 P bc5af9cdcf5d4b16b729c8ac3ceb361c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:40:02.338289 27528 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bc5af9cdcf5d4b16b729c8ac3ceb361c [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:40:02.339123 27528 raft_consensus.cc:504] T 00000000000000000000000000000000 P bc5af9cdcf5d4b16b729c8ac3ceb361c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc5af9cdcf5d4b16b729c8ac3ceb361c" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 35827 } }
I20260929 18:40:02.339447 27528 leader_election.cc:302] T 00000000000000000000000000000000 P bc5af9cdcf5d4b16b729c8ac3ceb361c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bc5af9cdcf5d4b16b729c8ac3ceb361c; no voters:
I20260929 18:40:02.339782 27528 leader_election.cc:288] T 00000000000000000000000000000000 P bc5af9cdcf5d4b16b729c8ac3ceb361c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:40:02.339843 27533 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bc5af9cdcf5d4b16b729c8ac3ceb361c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:40:02.340047 27533 raft_consensus.cc:686] T 00000000000000000000000000000000 P bc5af9cdcf5d4b16b729c8ac3ceb361c [term 1 LEADER]: Becoming Leader. State: Replica: bc5af9cdcf5d4b16b729c8ac3ceb361c, State: Running, Role: LEADER
I20260929 18:40:02.340390 27533 consensus_queue.cc:237] T 00000000000000000000000000000000 P bc5af9cdcf5d4b16b729c8ac3ceb361c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc5af9cdcf5d4b16b729c8ac3ceb361c" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 35827 } }
I20260929 18:40:02.341037 27528 sys_catalog.cc:564] T 00000000000000000000000000000000 P bc5af9cdcf5d4b16b729c8ac3ceb361c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:40:02.342036 27534 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc5af9cdcf5d4b16b729c8ac3ceb361c [sys.catalog]: SysCatalogTable state changed. Reason: New leader bc5af9cdcf5d4b16b729c8ac3ceb361c. Latest consensus state: current_term: 1 leader_uuid: "bc5af9cdcf5d4b16b729c8ac3ceb361c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc5af9cdcf5d4b16b729c8ac3ceb361c" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 35827 } } }
I20260929 18:40:02.342146 27534 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc5af9cdcf5d4b16b729c8ac3ceb361c [sys.catalog]: This master's current role is: LEADER
I20260929 18:40:02.342098 27535 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc5af9cdcf5d4b16b729c8ac3ceb361c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bc5af9cdcf5d4b16b729c8ac3ceb361c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc5af9cdcf5d4b16b729c8ac3ceb361c" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 35827 } } }
I20260929 18:40:02.342190 27535 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc5af9cdcf5d4b16b729c8ac3ceb361c [sys.catalog]: This master's current role is: LEADER
I20260929 18:40:02.342489 27542 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:40:02.342545 27531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:02.328727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55680 (local address 127.13.100.62:35827)
0929 18:40:02.329104 (+ 377us) server_negotiation.cc:206] Beginning negotiation
0929 18:40:02.329126 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0929 18:40:02.329161 (+ 35us) server_negotiation.cc:407] Connection header received
0929 18:40:02.329808 (+ 647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:40:02.329825 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:40:02.330209 (+ 384us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:40:02.330480 (+ 271us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:40:02.331337 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:02.332433 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:02.333540 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:02.334054 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:02.334167 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:40:02.337029 (+ 2862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:40:02.337034 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:40:02.337047 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:40:02.337074 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:40:02.338915 (+ 1841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:40:02.339351 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:40:02.339356 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:40:02.339362 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:40:02.339481 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:40:02.339751 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:40:02.339755 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:40:02.339757 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:40:02.340272 (+ 515us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:40:02.341004 (+ 732us) server_negotiation.cc:1035] Waiting for connection context
0929 18:40:02.341162 (+ 158us) server_negotiation.cc:299] Negotiation successful
0929 18:40:02.341465 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246,"thread_start_us":139,"threads_started":1}
I20260929 18:40:02.345646 27542 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:40:02.351517 27542 catalog_manager.cc:1347] Generated new cluster ID: b85dd72d97cd477f92a9a3802549ebfa
I20260929 18:40:02.351608 27542 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:40:02.361668 27542 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:40:02.362650 27542 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:40:02.369359 27542 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bc5af9cdcf5d4b16b729c8ac3ceb361c: Generated new TSK 0
I20260929 18:40:02.370111 27542 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.100.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.100.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.100.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.1.keytab.
Entry for principal kudu/127.13.100.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.1.keytab.
I20260929 18:40:02.402508 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.100.1:0
--local_ip_for_outbound_sockets=127.13.100.1
--webserver_interface=127.13.100.1
--webserver_port=0
--tserver_master_addrs=127.13.100.62:35827
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.1.keytab
--principal=kudu/127.13.100.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.100.20:43917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:40:02.521085 27554 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:40:02.521428 27554 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:40:02.521514 27554 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:40:02.525697 27554 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:40:02.525810 27554 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:40:02.525928 27554 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.100.1
I20260929 18:40:02.531466 27554 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.100.20:43917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.1.keytab
--principal=kudu/127.13.100.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.100.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.100.1
--webserver_port=0
--tserver_master_addrs=127.13.100.62:35827
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27554
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.100.1
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:40:02.532557 27554 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:40:02.533581 27554 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:40:02.543442 27559 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:40:02.543413 27560 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:40:02.543406 27562 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:40:02.543768 27554 server_base.cc:1034] running on GCE node
I20260929 18:40:02.544206 27554 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:40:02.545032 27554 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:40:02.546217 27554 hybrid_clock.cc:648] HybridClock initialized: now 1790707202546201 us; error 29 us; skew 500 ppm
Sep 29 18:40:02 dist-test-slave-j1xs krb5kdc[27447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707202, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:40:02.549007 27554 init.cc:377] Logged in from keytab as kudu/127.13.100.1@KRBTEST.COM (short username kudu)
I20260929 18:40:02.549919 27554 webserver.cc:460] Webserver started at http://127.13.100.1:38953/ using document root <none> and password file <none>
I20260929 18:40:02.550482 27554 fs_manager.cc:359] Metadata directory not provided
I20260929 18:40:02.550566 27554 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:40:02.550788 27554 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:40:02.552802 27554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/ts-0/data/instance:
uuid: "9fe6f60bc87f4337a740ea25d9265b13"
format_stamp: "Formatted at 2026-09-29 18:40:02 on dist-test-slave-j1xs"
server_key: "be8aa85caa3b213daed6b404d4b10721"
server_key_iv: "85d7fe3f3ffe0c297cd2dbbe62149376"
server_key_version: "encryptionkey@0"
I20260929 18:40:02.553373 27554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/ts-0/wal/instance:
uuid: "9fe6f60bc87f4337a740ea25d9265b13"
format_stamp: "Formatted at 2026-09-29 18:40:02 on dist-test-slave-j1xs"
server_key: "be8aa85caa3b213daed6b404d4b10721"
server_key_iv: "85d7fe3f3ffe0c297cd2dbbe62149376"
server_key_version: "encryptionkey@0"
I20260929 18:40:02.557044 27554 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260929 18:40:02.559484 27569 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:40:02.560655 27554 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 18:40:02.560786 27554 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/ts-0/wal
uuid: "9fe6f60bc87f4337a740ea25d9265b13"
format_stamp: "Formatted at 2026-09-29 18:40:02 on dist-test-slave-j1xs"
server_key: "be8aa85caa3b213daed6b404d4b10721"
server_key_iv: "85d7fe3f3ffe0c297cd2dbbe62149376"
server_key_version: "encryptionkey@0"
I20260929 18:40:02.560894 27554 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:40:02.579619 27554 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:40:02.582950 27554 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:40:02.583168 27554 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:40:02.583766 27554 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:40:02.584614 27554 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:40:02.584686 27554 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:40:02.584762 27554 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:40:02.584805 27554 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:40:02.594039 27554 rpc_server.cc:307] RPC server started. Bound to: 127.13.100.1:42269
I20260929 18:40:02.594063 27682 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.100.1:42269 every 8 connection(s)
I20260929 18:40:02.594921 27554 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/ts-0/data/info.pb
I20260929 18:40:02.599972 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 27554
I20260929 18:40:02.600116 13712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/ts-0/wal/instance
I20260929 18:40:02.600430 13712 external_mini_cluster.cc:1428] Setting key 94a0827680110b1784fc9e2efe9b2d0b
Sep 29 18:40:02 dist-test-slave-j1xs krb5kdc[27447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707202, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.1@KRBTEST.COM for kudu/127.13.100.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.13.100.2@KRBTEST.COM; defaulting to no policy
I20260929 18:40:02.609822 27531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:02.596720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.1:37291 (local address 127.13.100.62:35827)
0929 18:40:02.596886 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0929 18:40:02.596891 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:40:02.597709 (+ 818us) server_negotiation.cc:407] Connection header received
0929 18:40:02.599062 (+ 1353us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:40:02.599065 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:40:02.599117 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:40:02.599206 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:40:02.600687 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:02.601266 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:02.602922 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:02.603157 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:02.603246 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:40:02.605829 (+ 2583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:40:02.605837 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:40:02.605840 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:40:02.605870 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:40:02.607416 (+ 1546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:40:02.608286 (+ 870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:40:02.608289 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:40:02.608291 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:40:02.608379 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:40:02.608710 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:40:02.608715 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:40:02.608717 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:40:02.608906 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:40:02.609010 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 18:40:02.609533 (+ 523us) server_negotiation.cc:299] Negotiation successful
0929 18:40:02.609653 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 18:40:02.610788 27685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:02.597106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:35827 (local address 127.13.100.1:37291)
0929 18:40:02.597555 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0929 18:40:02.597599 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 18:40:02.598854 (+ 1255us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:40:02.599383 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:40:02.599390 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:40:02.599829 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:40:02.600538 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:40:02.600559 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:40:02.601390 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:40:02.601393 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:40:02.602791 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:40:02.602805 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:40:02.603306 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:40:02.603313 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:40:02.603555 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:40:02.604203 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:40:02.604219 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:40:02.605673 (+ 1454us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:40:02.607582 (+ 1909us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:40:02.607588 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:40:02.607603 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:40:02.608151 (+ 548us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:40:02.608470 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:40:02.608475 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:40:02.608477 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:40:02.608619 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:40:02.609009 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:40:02.609015 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:40:02.609290 (+ 275us) client_negotiation.cc:769] Sending connection context
0929 18:40:02.609494 (+ 204us) client_negotiation.cc:240] Negotiation successful
0929 18:40:02.609805 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":117,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.100.2@KRBTEST.COM" created.
I20260929 18:40:02.612003 27683 heartbeater.cc:347] Connected to a master server at 127.13.100.62:35827
I20260929 18:40:02.612247 27683 heartbeater.cc:464] Registering TS with master...
I20260929 18:40:02.612741 27683 heartbeater.cc:511] Master 127.13.100.62:35827 requested a full tablet report, sending...
I20260929 18:40:02.614369 27492 ts_manager.cc:194] Registered new tserver with Master: 9fe6f60bc87f4337a740ea25d9265b13 (127.13.100.1:42269)
I20260929 18:40:02.616019 27492 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.100.1@KRBTEST.COM'} at 127.13.100.1:37291
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.100.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.2.keytab.
Entry for principal kudu/127.13.100.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.2.keytab.
I20260929 18:40:02.628270 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.100.2:0
--local_ip_for_outbound_sockets=127.13.100.2
--webserver_interface=127.13.100.2
--webserver_port=0
--tserver_master_addrs=127.13.100.62:35827
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.2.keytab
--principal=kudu/127.13.100.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.100.20:43917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:40:02.737473 27688 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:40:02.737738 27688 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:40:02.737783 27688 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:40:02.741770 27688 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:40:02.741827 27688 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:40:02.741983 27688 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.100.2
I20260929 18:40:02.747390 27688 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.100.20:43917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.2.keytab
--principal=kudu/127.13.100.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.100.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.100.2
--webserver_port=0
--tserver_master_addrs=127.13.100.62:35827
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27688
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.100.2
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:40:02.748410 27688 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:40:02.749372 27688 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:40:02.759054 27696 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:40:02.759084 27693 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:40:02.759054 27694 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:40:02.759408 27688 server_base.cc:1034] running on GCE node
I20260929 18:40:02.759802 27688 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:40:02.760457 27688 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:40:02.761665 27688 hybrid_clock.cc:648] HybridClock initialized: now 1790707202761649 us; error 26 us; skew 500 ppm
Sep 29 18:40:02 dist-test-slave-j1xs krb5kdc[27447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707202, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:40:02.764482 27688 init.cc:377] Logged in from keytab as kudu/127.13.100.2@KRBTEST.COM (short username kudu)
I20260929 18:40:02.765386 27688 webserver.cc:460] Webserver started at http://127.13.100.2:46185/ using document root <none> and password file <none>
I20260929 18:40:02.765980 27688 fs_manager.cc:359] Metadata directory not provided
I20260929 18:40:02.766045 27688 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:40:02.766211 27688 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:40:02.768253 27688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/ts-1/data/instance:
uuid: "2b4519ad4d74462ebfa5eba155184701"
format_stamp: "Formatted at 2026-09-29 18:40:02 on dist-test-slave-j1xs"
server_key: "bce4f1c80ce470c76043a3749f9a2130"
server_key_iv: "c3668ad0ef7af0f4ace653617c9fb506"
server_key_version: "encryptionkey@0"
I20260929 18:40:02.768795 27688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/ts-1/wal/instance:
uuid: "2b4519ad4d74462ebfa5eba155184701"
format_stamp: "Formatted at 2026-09-29 18:40:02 on dist-test-slave-j1xs"
server_key: "bce4f1c80ce470c76043a3749f9a2130"
server_key_iv: "c3668ad0ef7af0f4ace653617c9fb506"
server_key_version: "encryptionkey@0"
I20260929 18:40:02.772302 27688 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260929 18:40:02.774672 27703 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:40:02.775876 27688 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:40:02.775969 27688 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/ts-1/wal
uuid: "2b4519ad4d74462ebfa5eba155184701"
format_stamp: "Formatted at 2026-09-29 18:40:02 on dist-test-slave-j1xs"
server_key: "bce4f1c80ce470c76043a3749f9a2130"
server_key_iv: "c3668ad0ef7af0f4ace653617c9fb506"
server_key_version: "encryptionkey@0"
I20260929 18:40:02.776063 27688 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:40:02.789383 27688 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:40:02.792388 27688 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:40:02.792546 27688 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:40:02.793068 27688 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:40:02.793833 27688 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:40:02.793879 27688 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:40:02.794004 27688 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:40:02.794028 27688 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:40:02.802732 27688 rpc_server.cc:307] RPC server started. Bound to: 127.13.100.2:44645
I20260929 18:40:02.802757 27816 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.100.2:44645 every 8 connection(s)
I20260929 18:40:02.803612 27688 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/ts-1/data/info.pb
I20260929 18:40:02.804441 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 27688
I20260929 18:40:02.804564 13712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/ts-1/wal/instance
I20260929 18:40:02.804839 13712 external_mini_cluster.cc:1428] Setting key 96cedbe226ce5aed4a69895eb5b00b1a
Sep 29 18:40:02 dist-test-slave-j1xs krb5kdc[27447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707202, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.2@KRBTEST.COM for kudu/127.13.100.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.13.100.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.100.3@KRBTEST.COM" created.
I20260929 18:40:02.818856 27531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:02.805396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.2:40775 (local address 127.13.100.62:35827)
0929 18:40:02.805549 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0929 18:40:02.805553 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:40:02.806500 (+ 947us) server_negotiation.cc:407] Connection header received
0929 18:40:02.807810 (+ 1310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:40:02.807813 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:40:02.807864 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:40:02.807941 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:40:02.809042 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:02.809921 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:02.811156 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:02.811604 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:02.811741 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:40:02.814310 (+ 2569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:40:02.814315 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:40:02.814317 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:40:02.814340 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:40:02.815756 (+ 1416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:40:02.816299 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:40:02.816308 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:40:02.816310 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:40:02.816396 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:40:02.816730 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:40:02.816733 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:40:02.816735 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:40:02.816967 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:40:02.817074 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 18:40:02.818620 (+ 1546us) server_negotiation.cc:299] Negotiation successful
0929 18:40:02.818724 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 18:40:02.819881 27819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:02.805820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:35827 (local address 127.13.100.2:40775)
0929 18:40:02.806339 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0929 18:40:02.806389 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 18:40:02.807615 (+ 1226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:40:02.808098 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:40:02.808104 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:40:02.808445 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:40:02.808906 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:40:02.808919 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:40:02.810051 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:40:02.810054 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:40:02.811027 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:40:02.811038 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:40:02.811785 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:40:02.811788 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:40:02.811947 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:40:02.812608 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:40:02.812624 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:40:02.814138 (+ 1514us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:40:02.815882 (+ 1744us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:40:02.815888 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:40:02.815903 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:40:02.816166 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:40:02.816492 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:40:02.816495 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:40:02.816497 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:40:02.816632 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:40:02.817993 (+ 1361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:40:02.817999 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:40:02.818310 (+ 311us) client_negotiation.cc:769] Sending connection context
0929 18:40:02.818553 (+ 243us) client_negotiation.cc:240] Negotiation successful
0929 18:40:02.818928 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":151,"threads_started":1}
I20260929 18:40:02.821077 27817 heartbeater.cc:347] Connected to a master server at 127.13.100.62:35827
I20260929 18:40:02.821342 27817 heartbeater.cc:464] Registering TS with master...
I20260929 18:40:02.821972 27817 heartbeater.cc:511] Master 127.13.100.62:35827 requested a full tablet report, sending...
I20260929 18:40:02.823145 27492 ts_manager.cc:194] Registered new tserver with Master: 2b4519ad4d74462ebfa5eba155184701 (127.13.100.2:44645)
I20260929 18:40:02.823963 27492 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.100.2@KRBTEST.COM'} at 127.13.100.2:40775
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.100.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.3.keytab.
Entry for principal kudu/127.13.100.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.3.keytab.
I20260929 18:40:02.833127 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.100.3:0
--local_ip_for_outbound_sockets=127.13.100.3
--webserver_interface=127.13.100.3
--webserver_port=0
--tserver_master_addrs=127.13.100.62:35827
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.3.keytab
--principal=kudu/127.13.100.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.100.20:43917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:40:02.944397 27822 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:40:02.944665 27822 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:40:02.944710 27822 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:40:02.948807 27822 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:40:02.948864 27822 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:40:02.948961 27822 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.100.3
I20260929 18:40:02.954288 27822 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.100.20:43917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.3.keytab
--principal=kudu/127.13.100.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.100.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.100.3
--webserver_port=0
--tserver_master_addrs=127.13.100.62:35827
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27822
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.100.3
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:40:02.955327 27822 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:40:02.956313 27822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:40:02.965869 27828 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:40:02.965884 27830 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:40:02.965854 27827 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:40:02.966061 27822 server_base.cc:1034] running on GCE node
I20260929 18:40:02.966457 27822 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:40:02.967068 27822 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:40:02.968245 27822 hybrid_clock.cc:648] HybridClock initialized: now 1790707202968212 us; error 46 us; skew 500 ppm
Sep 29 18:40:02 dist-test-slave-j1xs krb5kdc[27447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707202, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:40:02.970839 27822 init.cc:377] Logged in from keytab as kudu/127.13.100.3@KRBTEST.COM (short username kudu)
I20260929 18:40:02.971698 27822 webserver.cc:460] Webserver started at http://127.13.100.3:36755/ using document root <none> and password file <none>
I20260929 18:40:02.972240 27822 fs_manager.cc:359] Metadata directory not provided
I20260929 18:40:02.972327 27822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:40:02.972546 27822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:40:02.974578 27822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/ts-2/data/instance:
uuid: "a5aa041fdd8a4573b36ee7aee2085658"
format_stamp: "Formatted at 2026-09-29 18:40:02 on dist-test-slave-j1xs"
server_key: "351f8f834e1c4853a9ef0f872f19da00"
server_key_iv: "8a1dbdae6e6a05b37f329313c66820da"
server_key_version: "encryptionkey@0"
I20260929 18:40:02.975119 27822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/ts-2/wal/instance:
uuid: "a5aa041fdd8a4573b36ee7aee2085658"
format_stamp: "Formatted at 2026-09-29 18:40:02 on dist-test-slave-j1xs"
server_key: "351f8f834e1c4853a9ef0f872f19da00"
server_key_iv: "8a1dbdae6e6a05b37f329313c66820da"
server_key_version: "encryptionkey@0"
I20260929 18:40:02.978742 27822 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 18:40:02.981120 27837 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:40:02.982323 27822 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:40:02.982445 27822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/ts-2/wal
uuid: "a5aa041fdd8a4573b36ee7aee2085658"
format_stamp: "Formatted at 2026-09-29 18:40:02 on dist-test-slave-j1xs"
server_key: "351f8f834e1c4853a9ef0f872f19da00"
server_key_iv: "8a1dbdae6e6a05b37f329313c66820da"
server_key_version: "encryptionkey@0"
I20260929 18:40:02.982565 27822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:40:03.007913 27822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:40:03.010787 27822 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:40:03.010942 27822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:40:03.011469 27822 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:40:03.012224 27822 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:40:03.012269 27822 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:40:03.012308 27822 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:40:03.012329 27822 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:40:03.020993 27822 rpc_server.cc:307] RPC server started. Bound to: 127.13.100.3:45395
I20260929 18:40:03.021013 27950 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.100.3:45395 every 8 connection(s)
I20260929 18:40:03.021822 27822 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/ts-2/data/info.pb
I20260929 18:40:03.029623 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 27822
I20260929 18:40:03.029736 13712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/ts-2/wal/instance
I20260929 18:40:03.030053 13712 external_mini_cluster.cc:1428] Setting key 1f35a5a96436627983c525ad0533f02a
Sep 29 18:40:03 dist-test-slave-j1xs krb5kdc[27447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707202, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.3@KRBTEST.COM for kudu/127.13.100.62@KRBTEST.COM
I20260929 18:40:03.035106 27531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:03.023542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.3:52285 (local address 127.13.100.62:35827)
0929 18:40:03.023683 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0929 18:40:03.023686 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:40:03.024418 (+ 732us) server_negotiation.cc:407] Connection header received
0929 18:40:03.025754 (+ 1336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:40:03.025758 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:40:03.025832 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:40:03.025970 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:40:03.027076 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:03.027597 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:03.028696 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:03.028917 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:03.029022 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:40:03.031822 (+ 2800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:40:03.031826 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:40:03.031828 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:40:03.031854 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:40:03.033152 (+ 1298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:40:03.033632 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:40:03.033636 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:40:03.033638 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:40:03.033713 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:40:03.034112 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:40:03.034115 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:40:03.034117 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:40:03.034289 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:40:03.034393 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 18:40:03.034841 (+ 448us) server_negotiation.cc:299] Negotiation successful
0929 18:40:03.034953 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 18:40:03.035921 27953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:03.023836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:35827 (local address 127.13.100.3:52285)
0929 18:40:03.024259 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0929 18:40:03.024298 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 18:40:03.025544 (+ 1246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:40:03.026129 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:40:03.026135 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:40:03.026483 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:40:03.026932 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:40:03.026949 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:40:03.027719 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:40:03.027722 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:40:03.028572 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:40:03.028581 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:40:03.029063 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:40:03.029066 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:40:03.029227 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:40:03.029996 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:40:03.030012 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:40:03.031656 (+ 1644us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:40:03.033266 (+ 1610us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:40:03.033271 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:40:03.033285 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:40:03.033529 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:40:03.033808 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:40:03.033811 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:40:03.033812 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:40:03.034015 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:40:03.034390 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:40:03.034397 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:40:03.034624 (+ 227us) client_negotiation.cc:769] Sending connection context
0929 18:40:03.034800 (+ 176us) client_negotiation.cc:240] Negotiation successful
0929 18:40:03.035106 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":107,"threads_started":1}
I20260929 18:40:03.036972 27951 heartbeater.cc:347] Connected to a master server at 127.13.100.62:35827
I20260929 18:40:03.037177 27951 heartbeater.cc:464] Registering TS with master...
I20260929 18:40:03.037706 27951 heartbeater.cc:511] Master 127.13.100.62:35827 requested a full tablet report, sending...
I20260929 18:40:03.038715 27492 ts_manager.cc:194] Registered new tserver with Master: a5aa041fdd8a4573b36ee7aee2085658 (127.13.100.3:45395)
I20260929 18:40:03.039216 27492 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.100.3@KRBTEST.COM'} at 127.13.100.3:52285
I20260929 18:40:03.043565 13712 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 18:40:03.052024 27531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:03.045360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55690 (local address 127.13.100.62:35827)
0929 18:40:03.045465 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0929 18:40:03.045468 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:40:03.045675 (+ 207us) server_negotiation.cc:407] Connection header received
0929 18:40:03.045736 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:40:03.045738 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:40:03.045781 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:40:03.045849 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:40:03.046485 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:03.046977 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:03.047735 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:03.047945 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:03.048022 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:40:03.049193 (+ 1171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:40:03.049196 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:40:03.049198 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:40:03.049218 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:40:03.050648 (+ 1430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:40:03.051044 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:40:03.051046 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:40:03.051047 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:40:03.051119 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:40:03.051364 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:40:03.051366 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:40:03.051367 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:40:03.051556 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:40:03.051655 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0929 18:40:03.051822 (+ 167us) server_negotiation.cc:299] Negotiation successful
0929 18:40:03.051917 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 18:40:03.054641 27492 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55690:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:40:03.057544 27492 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:40:03.071686 27964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:03.065504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50430 (local address 127.13.100.3:45395)
0929 18:40:03.067197 (+ 1693us) server_negotiation.cc:206] Beginning negotiation
0929 18:40:03.067205 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 18:40:03.067221 (+ 16us) server_negotiation.cc:407] Connection header received
0929 18:40:03.067307 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:40:03.067312 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:40:03.067495 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:40:03.067672 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:40:03.067855 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:03.068960 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:03.070507 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:03.071046 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:03.071167 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:40:03.071308 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0929 18:40:03.071441 (+ 133us) server_negotiation.cc:299] Negotiation successful
0929 18:40:03.071533 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1616,"thread_start_us":131,"threads_started":1}
I20260929 18:40:03.071677 27963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:03.065337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.3:45395 (local address 127.0.0.1:50430)
0929 18:40:03.066666 (+ 1329us) negotiation.cc:107] Waiting for socket to connect
0929 18:40:03.066701 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0929 18:40:03.066796 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:40:03.067645 (+ 849us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:40:03.067648 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:40:03.067664 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:40:03.067749 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:40:03.067755 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:40:03.069118 (+ 1363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:40:03.069121 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:40:03.070376 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:40:03.070387 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:40:03.071165 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:40:03.071169 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:40:03.071270 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:40:03.071304 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 18:40:03.071416 (+ 112us) client_negotiation.cc:240] Negotiation successful
0929 18:40:03.071533 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1159,"mutex_wait_us":24,"thread_start_us":103,"threads_started":1}
I20260929 18:40:03.071902 27965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:03.065532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.2:44645 (local address 127.0.0.1:36408)
0929 18:40:03.066667 (+ 1135us) negotiation.cc:107] Waiting for socket to connect
0929 18:40:03.066703 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 18:40:03.066801 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:40:03.067326 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:40:03.067332 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:40:03.067355 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:40:03.067480 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:40:03.067495 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:40:03.068640 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:40:03.068647 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:40:03.070327 (+ 1680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:40:03.070338 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:40:03.071354 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:40:03.071358 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:40:03.071448 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:40:03.071472 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 18:40:03.071536 (+ 64us) client_negotiation.cc:240] Negotiation successful
0929 18:40:03.071606 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1041,"mutex_wait_us":35,"spinlock_wait_cycles":768,"thread_start_us":98,"threads_started":1}
I20260929 18:40:03.072024 27962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:03.066104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.1:42269 (local address 127.0.0.1:48242)
0929 18:40:03.066950 (+ 846us) negotiation.cc:107] Waiting for socket to connect
0929 18:40:03.066959 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 18:40:03.067020 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:40:03.067578 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:40:03.067583 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:40:03.067602 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:40:03.067696 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:40:03.067702 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:40:03.069363 (+ 1661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:40:03.069367 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:40:03.070201 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:40:03.070210 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:40:03.071474 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:40:03.071477 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:40:03.071564 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:40:03.071581 (+ 17us) client_negotiation.cc:769] Sending connection context
0929 18:40:03.071640 (+ 59us) client_negotiation.cc:240] Negotiation successful
0929 18:40:03.071699 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":803,"thread_start_us":99,"threads_started":1}
I20260929 18:40:03.072175 27966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:03.066133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36408 (local address 127.13.100.2:44645)
0929 18:40:03.066786 (+ 653us) server_negotiation.cc:206] Beginning negotiation
0929 18:40:03.066798 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 18:40:03.066821 (+ 23us) server_negotiation.cc:407] Connection header received
0929 18:40:03.066949 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:40:03.066954 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:40:03.067171 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:40:03.067323 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:40:03.067619 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:03.068524 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:03.070460 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:03.071228 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:03.071743 (+ 515us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:40:03.071880 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0929 18:40:03.071930 (+ 50us) server_negotiation.cc:299] Negotiation successful
0929 18:40:03.072020 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":549,"thread_start_us":101,"threads_started":1}
I20260929 18:40:03.072439 27967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:03.066393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48242 (local address 127.13.100.1:42269)
0929 18:40:03.067027 (+ 634us) server_negotiation.cc:206] Beginning negotiation
0929 18:40:03.067045 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 18:40:03.067066 (+ 21us) server_negotiation.cc:407] Connection header received
0929 18:40:03.067129 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:40:03.067133 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:40:03.067287 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:40:03.067428 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:40:03.068014 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:03.069213 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:03.070324 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:03.071159 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:03.071282 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:40:03.071423 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0929 18:40:03.072197 (+ 774us) server_negotiation.cc:299] Negotiation successful
0929 18:40:03.072303 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":528,"thread_start_us":91,"threads_started":1}
I20260929 18:40:03.075131 27885 tablet_service.cc:1467] Processing CreateTablet for tablet 1338ed01901440118fbcca2a287b7ff8 (DEFAULT_TABLE table=test-table [id=00fdaf3ea6e74809935ca13bf796543c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:40:03.075191 27751 tablet_service.cc:1467] Processing CreateTablet for tablet 1338ed01901440118fbcca2a287b7ff8 (DEFAULT_TABLE table=test-table [id=00fdaf3ea6e74809935ca13bf796543c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:40:03.075191 27617 tablet_service.cc:1467] Processing CreateTablet for tablet 1338ed01901440118fbcca2a287b7ff8 (DEFAULT_TABLE table=test-table [id=00fdaf3ea6e74809935ca13bf796543c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:40:03.076126 27885 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1338ed01901440118fbcca2a287b7ff8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:40:03.076184 27751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1338ed01901440118fbcca2a287b7ff8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:40:03.076184 27617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1338ed01901440118fbcca2a287b7ff8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:40:03.082403 27968 tablet_bootstrap.cc:492] T 1338ed01901440118fbcca2a287b7ff8 P a5aa041fdd8a4573b36ee7aee2085658: Bootstrap starting.
I20260929 18:40:03.083770 27969 tablet_bootstrap.cc:492] T 1338ed01901440118fbcca2a287b7ff8 P 2b4519ad4d74462ebfa5eba155184701: Bootstrap starting.
I20260929 18:40:03.085530 27968 tablet_bootstrap.cc:654] T 1338ed01901440118fbcca2a287b7ff8 P a5aa041fdd8a4573b36ee7aee2085658: Neither blocks nor log segments found. Creating new log.
I20260929 18:40:03.085784 27969 tablet_bootstrap.cc:654] T 1338ed01901440118fbcca2a287b7ff8 P 2b4519ad4d74462ebfa5eba155184701: Neither blocks nor log segments found. Creating new log.
I20260929 18:40:03.086223 27970 tablet_bootstrap.cc:492] T 1338ed01901440118fbcca2a287b7ff8 P 9fe6f60bc87f4337a740ea25d9265b13: Bootstrap starting.
I20260929 18:40:03.086385 27968 log.cc:824] T 1338ed01901440118fbcca2a287b7ff8 P a5aa041fdd8a4573b36ee7aee2085658: Log is configured to *not* fsync() on all Append() calls
I20260929 18:40:03.086616 27969 log.cc:824] T 1338ed01901440118fbcca2a287b7ff8 P 2b4519ad4d74462ebfa5eba155184701: Log is configured to *not* fsync() on all Append() calls
I20260929 18:40:03.088371 27970 tablet_bootstrap.cc:654] T 1338ed01901440118fbcca2a287b7ff8 P 9fe6f60bc87f4337a740ea25d9265b13: Neither blocks nor log segments found. Creating new log.
I20260929 18:40:03.088474 27968 tablet_bootstrap.cc:492] T 1338ed01901440118fbcca2a287b7ff8 P a5aa041fdd8a4573b36ee7aee2085658: No bootstrap required, opened a new log
I20260929 18:40:03.088408 27969 tablet_bootstrap.cc:492] T 1338ed01901440118fbcca2a287b7ff8 P 2b4519ad4d74462ebfa5eba155184701: No bootstrap required, opened a new log
I20260929 18:40:03.088682 27968 ts_tablet_manager.cc:1397] T 1338ed01901440118fbcca2a287b7ff8 P a5aa041fdd8a4573b36ee7aee2085658: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 18:40:03.088802 27969 ts_tablet_manager.cc:1397] T 1338ed01901440118fbcca2a287b7ff8 P 2b4519ad4d74462ebfa5eba155184701: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260929 18:40:03.089607 27970 log.cc:824] T 1338ed01901440118fbcca2a287b7ff8 P 9fe6f60bc87f4337a740ea25d9265b13: Log is configured to *not* fsync() on all Append() calls
I20260929 18:40:03.091617 27970 tablet_bootstrap.cc:492] T 1338ed01901440118fbcca2a287b7ff8 P 9fe6f60bc87f4337a740ea25d9265b13: No bootstrap required, opened a new log
I20260929 18:40:03.091800 27970 ts_tablet_manager.cc:1397] T 1338ed01901440118fbcca2a287b7ff8 P 9fe6f60bc87f4337a740ea25d9265b13: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20260929 18:40:03.092064 27969 raft_consensus.cc:348] T 1338ed01901440118fbcca2a287b7ff8 P 2b4519ad4d74462ebfa5eba155184701 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5aa041fdd8a4573b36ee7aee2085658" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 45395 } } peers { permanent_uuid: "2b4519ad4d74462ebfa5eba155184701" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 44645 } } peers { permanent_uuid: "9fe6f60bc87f4337a740ea25d9265b13" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 42269 } }
I20260929 18:40:03.092288 27969 raft_consensus.cc:374] T 1338ed01901440118fbcca2a287b7ff8 P 2b4519ad4d74462ebfa5eba155184701 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:40:03.092336 27969 raft_consensus.cc:729] T 1338ed01901440118fbcca2a287b7ff8 P 2b4519ad4d74462ebfa5eba155184701 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b4519ad4d74462ebfa5eba155184701, State: Initialized, Role: FOLLOWER
I20260929 18:40:03.092439 27968 raft_consensus.cc:348] T 1338ed01901440118fbcca2a287b7ff8 P a5aa041fdd8a4573b36ee7aee2085658 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5aa041fdd8a4573b36ee7aee2085658" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 45395 } } peers { permanent_uuid: "2b4519ad4d74462ebfa5eba155184701" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 44645 } } peers { permanent_uuid: "9fe6f60bc87f4337a740ea25d9265b13" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 42269 } }
I20260929 18:40:03.092684 27968 raft_consensus.cc:374] T 1338ed01901440118fbcca2a287b7ff8 P a5aa041fdd8a4573b36ee7aee2085658 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:40:03.092751 27968 raft_consensus.cc:729] T 1338ed01901440118fbcca2a287b7ff8 P a5aa041fdd8a4573b36ee7aee2085658 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5aa041fdd8a4573b36ee7aee2085658, State: Initialized, Role: FOLLOWER
I20260929 18:40:03.092834 27969 consensus_queue.cc:260] T 1338ed01901440118fbcca2a287b7ff8 P 2b4519ad4d74462ebfa5eba155184701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5aa041fdd8a4573b36ee7aee2085658" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 45395 } } peers { permanent_uuid: "2b4519ad4d74462ebfa5eba155184701" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 44645 } } peers { permanent_uuid: "9fe6f60bc87f4337a740ea25d9265b13" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 42269 } }
I20260929 18:40:03.093163 27968 consensus_queue.cc:260] T 1338ed01901440118fbcca2a287b7ff8 P a5aa041fdd8a4573b36ee7aee2085658 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5aa041fdd8a4573b36ee7aee2085658" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 45395 } } peers { permanent_uuid: "2b4519ad4d74462ebfa5eba155184701" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 44645 } } peers { permanent_uuid: "9fe6f60bc87f4337a740ea25d9265b13" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 42269 } }
I20260929 18:40:03.093560 27817 heartbeater.cc:503] Master 127.13.100.62:35827 was elected leader, sending a full tablet report...
I20260929 18:40:03.093833 27969 ts_tablet_manager.cc:1428] T 1338ed01901440118fbcca2a287b7ff8 P 2b4519ad4d74462ebfa5eba155184701: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260929 18:40:03.094092 27968 ts_tablet_manager.cc:1428] T 1338ed01901440118fbcca2a287b7ff8 P a5aa041fdd8a4573b36ee7aee2085658: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
I20260929 18:40:03.094429 27951 heartbeater.cc:503] Master 127.13.100.62:35827 was elected leader, sending a full tablet report...
I20260929 18:40:03.095507 27970 raft_consensus.cc:348] T 1338ed01901440118fbcca2a287b7ff8 P 9fe6f60bc87f4337a740ea25d9265b13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5aa041fdd8a4573b36ee7aee2085658" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 45395 } } peers { permanent_uuid: "2b4519ad4d74462ebfa5eba155184701" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 44645 } } peers { permanent_uuid: "9fe6f60bc87f4337a740ea25d9265b13" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 42269 } }
I20260929 18:40:03.095743 27970 raft_consensus.cc:374] T 1338ed01901440118fbcca2a287b7ff8 P 9fe6f60bc87f4337a740ea25d9265b13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:40:03.095803 27970 raft_consensus.cc:729] T 1338ed01901440118fbcca2a287b7ff8 P 9fe6f60bc87f4337a740ea25d9265b13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fe6f60bc87f4337a740ea25d9265b13, State: Initialized, Role: FOLLOWER
I20260929 18:40:03.096380 27970 consensus_queue.cc:260] T 1338ed01901440118fbcca2a287b7ff8 P 9fe6f60bc87f4337a740ea25d9265b13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5aa041fdd8a4573b36ee7aee2085658" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 45395 } } peers { permanent_uuid: "2b4519ad4d74462ebfa5eba155184701" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 44645 } } peers { permanent_uuid: "9fe6f60bc87f4337a740ea25d9265b13" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 42269 } }
I20260929 18:40:03.097275 27970 ts_tablet_manager.cc:1428] T 1338ed01901440118fbcca2a287b7ff8 P 9fe6f60bc87f4337a740ea25d9265b13: Time spent starting tablet: real 0.005s user 0.004s sys 0.001s
I20260929 18:40:03.097596 27683 heartbeater.cc:503] Master 127.13.100.62:35827 was elected leader, sending a full tablet report...
W20260929 18:40:03.272789 27952 tablet.cc:2369] T 1338ed01901440118fbcca2a287b7ff8 P a5aa041fdd8a4573b36ee7aee2085658: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:40:03.297097 27976 raft_consensus.cc:482] T 1338ed01901440118fbcca2a287b7ff8 P 9fe6f60bc87f4337a740ea25d9265b13 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:40:03.297374 27976 raft_consensus.cc:504] T 1338ed01901440118fbcca2a287b7ff8 P 9fe6f60bc87f4337a740ea25d9265b13 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5aa041fdd8a4573b36ee7aee2085658" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 45395 } } peers { permanent_uuid: "2b4519ad4d74462ebfa5eba155184701" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 44645 } } peers { permanent_uuid: "9fe6f60bc87f4337a740ea25d9265b13" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 42269 } }
I20260929 18:40:03.298720 27976 leader_election.cc:288] T 1338ed01901440118fbcca2a287b7ff8 P 9fe6f60bc87f4337a740ea25d9265b13 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a5aa041fdd8a4573b36ee7aee2085658 (127.13.100.3:45395), 2b4519ad4d74462ebfa5eba155184701 (127.13.100.2:44645)
I20260929 18:40:03.303018 27966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:03.299174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.1:56703 (local address 127.13.100.2:44645)
0929 18:40:03.299306 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 18:40:03.299312 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:40:03.299772 (+ 460us) server_negotiation.cc:407] Connection header received
0929 18:40:03.299975 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:40:03.299981 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:40:03.300036 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:40:03.300110 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:40:03.300398 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:03.300876 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:03.302066 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:03.302451 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:03.302583 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:40:03.302637 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0929 18:40:03.302809 (+ 172us) server_negotiation.cc:299] Negotiation successful
0929 18:40:03.302877 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 18:40:03.303071 27964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:03.299090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.1:56329 (local address 127.13.100.3:45395)
0929 18:40:03.299243 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0929 18:40:03.299249 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:40:03.299360 (+ 111us) server_negotiation.cc:407] Connection header received
0929 18:40:03.299540 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:40:03.299546 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:40:03.299614 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:40:03.299683 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:40:03.300109 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:03.300730 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:03.302039 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:03.302460 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:03.302601 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:40:03.302663 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0929 18:40:03.302861 (+ 198us) server_negotiation.cc:299] Negotiation successful
0929 18:40:03.302940 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 18:40:03.303361 27978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:03.298981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.2:44645 (local address 127.13.100.1:56703)
0929 18:40:03.299693 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0929 18:40:03.299707 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:40:03.299842 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:40:03.300120 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:40:03.300123 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:40:03.300142 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:40:03.300258 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:40:03.300265 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:40:03.300986 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:40:03.300993 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:40:03.301882 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:40:03.301915 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:40:03.302576 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:40:03.302580 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:40:03.302681 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:40:03.302704 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 18:40:03.302794 (+ 90us) client_negotiation.cc:240] Negotiation successful
0929 18:40:03.302866 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":632,"mutex_wait_us":301,"thread_start_us":127,"threads_started":1}
I20260929 18:40:03.303442 27977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:03.298981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.3:45395 (local address 127.13.100.1:56329)
0929 18:40:03.299265 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0929 18:40:03.299282 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 18:40:03.299404 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:40:03.299741 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:40:03.299744 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:40:03.299768 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:40:03.299935 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:40:03.299947 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:40:03.300861 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:40:03.300867 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:40:03.301825 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:40:03.301836 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:40:03.302593 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:40:03.302597 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:40:03.302715 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:40:03.302735 (+ 20us) client_negotiation.cc:769] Sending connection context
0929 18:40:03.302827 (+ 92us) client_negotiation.cc:240] Negotiation successful
0929 18:40:03.302906 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"mutex_wait_us":50,"thread_start_us":84,"threads_started":1}
I20260929 18:40:03.303982 27771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1338ed01901440118fbcca2a287b7ff8" candidate_uuid: "9fe6f60bc87f4337a740ea25d9265b13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b4519ad4d74462ebfa5eba155184701" is_pre_election: true
I20260929 18:40:03.304091 27905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1338ed01901440118fbcca2a287b7ff8" candidate_uuid: "9fe6f60bc87f4337a740ea25d9265b13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5aa041fdd8a4573b36ee7aee2085658" is_pre_election: true
I20260929 18:40:03.304328 27771 raft_consensus.cc:2454] T 1338ed01901440118fbcca2a287b7ff8 P 2b4519ad4d74462ebfa5eba155184701 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9fe6f60bc87f4337a740ea25d9265b13 in term 0.
I20260929 18:40:03.304405 27905 raft_consensus.cc:2454] T 1338ed01901440118fbcca2a287b7ff8 P a5aa041fdd8a4573b36ee7aee2085658 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9fe6f60bc87f4337a740ea25d9265b13 in term 0.
W20260929 18:40:03.304594 27818 tablet.cc:2369] T 1338ed01901440118fbcca2a287b7ff8 P 2b4519ad4d74462ebfa5eba155184701: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:40:03.304858 27573 leader_election.cc:302] T 1338ed01901440118fbcca2a287b7ff8 P 9fe6f60bc87f4337a740ea25d9265b13 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b4519ad4d74462ebfa5eba155184701, 9fe6f60bc87f4337a740ea25d9265b13; no voters:
I20260929 18:40:03.305150 27976 raft_consensus.cc:2781] T 1338ed01901440118fbcca2a287b7ff8 P 9fe6f60bc87f4337a740ea25d9265b13 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:40:03.305253 27976 raft_consensus.cc:482] T 1338ed01901440118fbcca2a287b7ff8 P 9fe6f60bc87f4337a740ea25d9265b13 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:40:03.305311 27976 raft_consensus.cc:3037] T 1338ed01901440118fbcca2a287b7ff8 P 9fe6f60bc87f4337a740ea25d9265b13 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:40:03.306416 27976 raft_consensus.cc:504] T 1338ed01901440118fbcca2a287b7ff8 P 9fe6f60bc87f4337a740ea25d9265b13 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5aa041fdd8a4573b36ee7aee2085658" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 45395 } } peers { permanent_uuid: "2b4519ad4d74462ebfa5eba155184701" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 44645 } } peers { permanent_uuid: "9fe6f60bc87f4337a740ea25d9265b13" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 42269 } }
I20260929 18:40:03.306800 27976 leader_election.cc:288] T 1338ed01901440118fbcca2a287b7ff8 P 9fe6f60bc87f4337a740ea25d9265b13 [CANDIDATE]: Term 1 election: Requested vote from peers a5aa041fdd8a4573b36ee7aee2085658 (127.13.100.3:45395), 2b4519ad4d74462ebfa5eba155184701 (127.13.100.2:44645)
I20260929 18:40:03.307145 27905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1338ed01901440118fbcca2a287b7ff8" candidate_uuid: "9fe6f60bc87f4337a740ea25d9265b13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5aa041fdd8a4573b36ee7aee2085658"
I20260929 18:40:03.307196 27771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1338ed01901440118fbcca2a287b7ff8" candidate_uuid: "9fe6f60bc87f4337a740ea25d9265b13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b4519ad4d74462ebfa5eba155184701"
I20260929 18:40:03.307272 27905 raft_consensus.cc:3037] T 1338ed01901440118fbcca2a287b7ff8 P a5aa041fdd8a4573b36ee7aee2085658 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:40:03.307322 27771 raft_consensus.cc:3037] T 1338ed01901440118fbcca2a287b7ff8 P 2b4519ad4d74462ebfa5eba155184701 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:40:03.308295 27905 raft_consensus.cc:2454] T 1338ed01901440118fbcca2a287b7ff8 P a5aa041fdd8a4573b36ee7aee2085658 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9fe6f60bc87f4337a740ea25d9265b13 in term 1.
I20260929 18:40:03.308324 27771 raft_consensus.cc:2454] T 1338ed01901440118fbcca2a287b7ff8 P 2b4519ad4d74462ebfa5eba155184701 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9fe6f60bc87f4337a740ea25d9265b13 in term 1.
I20260929 18:40:03.308574 27571 leader_election.cc:302] T 1338ed01901440118fbcca2a287b7ff8 P 9fe6f60bc87f4337a740ea25d9265b13 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9fe6f60bc87f4337a740ea25d9265b13, a5aa041fdd8a4573b36ee7aee2085658; no voters:
I20260929 18:40:03.308753 27976 raft_consensus.cc:2781] T 1338ed01901440118fbcca2a287b7ff8 P 9fe6f60bc87f4337a740ea25d9265b13 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:40:03.308990 27976 raft_consensus.cc:686] T 1338ed01901440118fbcca2a287b7ff8 P 9fe6f60bc87f4337a740ea25d9265b13 [term 1 LEADER]: Becoming Leader. State: Replica: 9fe6f60bc87f4337a740ea25d9265b13, State: Running, Role: LEADER
I20260929 18:40:03.309293 27976 consensus_queue.cc:237] T 1338ed01901440118fbcca2a287b7ff8 P 9fe6f60bc87f4337a740ea25d9265b13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5aa041fdd8a4573b36ee7aee2085658" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 45395 } } peers { permanent_uuid: "2b4519ad4d74462ebfa5eba155184701" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 44645 } } peers { permanent_uuid: "9fe6f60bc87f4337a740ea25d9265b13" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 42269 } }
I20260929 18:40:03.312252 27491 catalog_manager.cc:5495] T 1338ed01901440118fbcca2a287b7ff8 P 9fe6f60bc87f4337a740ea25d9265b13 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9fe6f60bc87f4337a740ea25d9265b13 (127.13.100.1). New cstate: current_term: 1 leader_uuid: "9fe6f60bc87f4337a740ea25d9265b13" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5aa041fdd8a4573b36ee7aee2085658" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 45395 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b4519ad4d74462ebfa5eba155184701" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 44645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9fe6f60bc87f4337a740ea25d9265b13" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 42269 } health_report { overall_health: HEALTHY } } }
W20260929 18:40:03.346207 27684 tablet.cc:2369] T 1338ed01901440118fbcca2a287b7ff8 P 9fe6f60bc87f4337a740ea25d9265b13: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:40:03.366014 27967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:03.362156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48252 (local address 127.13.100.1:42269)
0929 18:40:03.362277 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0929 18:40:03.362281 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:40:03.362294 (+ 13us) server_negotiation.cc:407] Connection header received
0929 18:40:03.362391 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:40:03.362393 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:40:03.362449 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:40:03.362545 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:40:03.362858 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:03.363383 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:03.364324 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:03.364611 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:03.364719 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:40:03.365062 (+ 343us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:40:03.365567 (+ 505us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:40:03.365660 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 18:40:03.365825 (+ 165us) server_negotiation.cc:299] Negotiation successful
0929 18:40:03.365879 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 18:40:03.371959 27905 raft_consensus.cc:1261] T 1338ed01901440118fbcca2a287b7ff8 P a5aa041fdd8a4573b36ee7aee2085658 [term 1 FOLLOWER]: Refusing update from remote peer 9fe6f60bc87f4337a740ea25d9265b13: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:40:03.371959 27771 raft_consensus.cc:1261] T 1338ed01901440118fbcca2a287b7ff8 P 2b4519ad4d74462ebfa5eba155184701 [term 1 FOLLOWER]: Refusing update from remote peer 9fe6f60bc87f4337a740ea25d9265b13: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:40:03.372807 27976 consensus_queue.cc:1035] T 1338ed01901440118fbcca2a287b7ff8 P 9fe6f60bc87f4337a740ea25d9265b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5aa041fdd8a4573b36ee7aee2085658" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 45395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:40:03.373018 27979 consensus_queue.cc:1035] T 1338ed01901440118fbcca2a287b7ff8 P 9fe6f60bc87f4337a740ea25d9265b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b4519ad4d74462ebfa5eba155184701" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:40:03.389729 27492 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55690:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:40:03.389927 27492 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55690:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:40:03.392210 27492 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bc5af9cdcf5d4b16b729c8ac3ceb361c: Sending DeleteTablet for 3 replicas of tablet 1338ed01901440118fbcca2a287b7ff8
I20260929 18:40:03.392957 27885 tablet_service.cc:1514] Processing DeleteTablet for tablet 1338ed01901440118fbcca2a287b7ff8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:40:03 UTC) from {username='kudu', principal='kudu/127.13.100.62@KRBTEST.COM'} at 127.0.0.1:50430
I20260929 18:40:03.393005 27751 tablet_service.cc:1514] Processing DeleteTablet for tablet 1338ed01901440118fbcca2a287b7ff8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:40:03 UTC) from {username='kudu', principal='kudu/127.13.100.62@KRBTEST.COM'} at 127.0.0.1:36408
I20260929 18:40:03.393227 27617 tablet_service.cc:1514] Processing DeleteTablet for tablet 1338ed01901440118fbcca2a287b7ff8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:40:03 UTC) from {username='kudu', principal='kudu/127.13.100.62@KRBTEST.COM'} at 127.0.0.1:48242
I20260929 18:40:03.393676 27994 tablet_replica.cc:330] T 1338ed01901440118fbcca2a287b7ff8 P 9fe6f60bc87f4337a740ea25d9265b13: stopping tablet replica
I20260929 18:40:03.393801 27993 tablet_replica.cc:330] T 1338ed01901440118fbcca2a287b7ff8 P 2b4519ad4d74462ebfa5eba155184701: stopping tablet replica
I20260929 18:40:03.393923 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 27554
I20260929 18:40:03.394099 27994 raft_consensus.cc:2229] T 1338ed01901440118fbcca2a287b7ff8 P 9fe6f60bc87f4337a740ea25d9265b13 [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:40:03.395076 27993 raft_consensus.cc:2229] T 1338ed01901440118fbcca2a287b7ff8 P 2b4519ad4d74462ebfa5eba155184701 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:40:03.395421 27993 raft_consensus.cc:2258] T 1338ed01901440118fbcca2a287b7ff8 P 2b4519ad4d74462ebfa5eba155184701 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:40:03.395803 27992 tablet_replica.cc:330] T 1338ed01901440118fbcca2a287b7ff8 P a5aa041fdd8a4573b36ee7aee2085658: stopping tablet replica
I20260929 18:40:03.396088 27992 raft_consensus.cc:2229] T 1338ed01901440118fbcca2a287b7ff8 P a5aa041fdd8a4573b36ee7aee2085658 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:40:03.396355 27992 raft_consensus.cc:2258] T 1338ed01901440118fbcca2a287b7ff8 P a5aa041fdd8a4573b36ee7aee2085658 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:40:03.397107 27993 ts_tablet_manager.cc:1905] T 1338ed01901440118fbcca2a287b7ff8 P 2b4519ad4d74462ebfa5eba155184701: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:40:03.397567 27992 ts_tablet_manager.cc:1905] T 1338ed01901440118fbcca2a287b7ff8 P a5aa041fdd8a4573b36ee7aee2085658: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:40:03.399828 27992 ts_tablet_manager.cc:1918] T 1338ed01901440118fbcca2a287b7ff8 P a5aa041fdd8a4573b36ee7aee2085658: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:40:03.399938 27992 log.cc:1196] T 1338ed01901440118fbcca2a287b7ff8 P a5aa041fdd8a4573b36ee7aee2085658: Deleting WAL directory at /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/ts-2/wal/wals/1338ed01901440118fbcca2a287b7ff8
I20260929 18:40:03.400223 27992 ts_tablet_manager.cc:1939] T 1338ed01901440118fbcca2a287b7ff8 P a5aa041fdd8a4573b36ee7aee2085658: Deleting consensus metadata
I20260929 18:40:03.400537 27993 ts_tablet_manager.cc:1918] T 1338ed01901440118fbcca2a287b7ff8 P 2b4519ad4d74462ebfa5eba155184701: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:40:03.400641 27993 log.cc:1196] T 1338ed01901440118fbcca2a287b7ff8 P 2b4519ad4d74462ebfa5eba155184701: Deleting WAL directory at /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/ts-1/wal/wals/1338ed01901440118fbcca2a287b7ff8
I20260929 18:40:03.400950 27993 ts_tablet_manager.cc:1939] T 1338ed01901440118fbcca2a287b7ff8 P 2b4519ad4d74462ebfa5eba155184701: Deleting consensus metadata
I20260929 18:40:03.401517 27477 catalog_manager.cc:4841] TS a5aa041fdd8a4573b36ee7aee2085658 (127.13.100.3:45395): tablet 1338ed01901440118fbcca2a287b7ff8 (table test-table [id=00fdaf3ea6e74809935ca13bf796543c]) successfully deleted
W20260929 18:40:03.401645 27476 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.100.1:42269 (error 108)
I20260929 18:40:03.401844 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 27688
I20260929 18:40:03.403306 27962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:03.402935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.1:42269 (local address 127.0.0.1:48262)
0929 18:40:03.403115 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0929 18:40:03.403197 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.1:42269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260929 18:40:03.403824 27476 catalog_manager.cc:4584] TS 9fe6f60bc87f4337a740ea25d9265b13 (127.13.100.1:42269): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1338ed01901440118fbcca2a287b7ff8: Network error: Client connection negotiation failed: client connection to 127.13.100.1:42269: connect: Connection refused (error 111)
I20260929 18:40:03.408139 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 27822
I20260929 18:40:03.409085 27962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:03.408310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.2:44645 (local address 127.0.0.1:36420)
0929 18:40:03.408917 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0929 18:40:03.408982 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.2:44645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":508}
I20260929 18:40:03.414454 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 27460
I20260929 18:40:03.421057 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.100.62:35827
--webserver_interface=127.13.100.62
--webserver_port=46467
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.62.keytab
--principal=kudu/127.13.100.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.100.20:43917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.100.62:35827
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:40:03.528839 27996 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:40:03.529093 27996 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:40:03.529134 27996 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:40:03.533859 27996 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:40:03.533936 27996 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:40:03.533972 27996 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:40:03.534009 27996 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:40:03.534044 27996 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:40:03.539537 27996 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.100.20:43917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.13.100.62:35827
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.62.keytab
--principal=kudu/127.13.100.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.100.62:35827
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.100.62
--webserver_port=46467
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27996
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:40:03.540566 27996 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:40:03.541570 27996 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:40:03.554354 28005 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:40:03.554354 28003 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:40:03.554370 28002 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:40:03.554497 27996 server_base.cc:1034] running on GCE node
I20260929 18:40:03.555171 27996 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:40:03.556358 27996 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:40:03.557521 27996 hybrid_clock.cc:648] HybridClock initialized: now 1790707203557501 us; error 34 us; skew 500 ppm
Sep 29 18:40:03 dist-test-slave-j1xs krb5kdc[27447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707203, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:40:03.560359 27996 init.cc:377] Logged in from keytab as kudu/127.13.100.62@KRBTEST.COM (short username kudu)
I20260929 18:40:03.561326 27996 webserver.cc:460] Webserver started at http://127.13.100.62:46467/ using document root <none> and password file <none>
I20260929 18:40:03.561971 27996 fs_manager.cc:359] Metadata directory not provided
I20260929 18:40:03.562074 27996 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:40:03.566025 27996 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 18:40:03.568307 28012 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:40:03.569948 27996 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:40:03.570086 27996 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/master-0/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/master-0/wal
uuid: "bc5af9cdcf5d4b16b729c8ac3ceb361c"
format_stamp: "Formatted at 2026-09-29 18:40:02 on dist-test-slave-j1xs"
server_key: "f78f34ab993a73e964650e60e2595883"
server_key_iv: "29027fd7bcfe9d91d74d334f1f49ab83"
server_key_version: "encryptionkey@0"
I20260929 18:40:03.570701 27996 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:40:03.589650 27996 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:40:03.592741 27996 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:40:03.592967 27996 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:40:03.600400 27996 rpc_server.cc:307] RPC server started. Bound to: 127.13.100.62:35827
I20260929 18:40:03.600853 28064 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.100.62:35827 every 8 connection(s)
I20260929 18:40:03.601531 27996 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
I20260929 18:40:03.605816 28065 sys_catalog.cc:263] Verifying existing consensus state
I20260929 18:40:03.607195 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 27996
I20260929 18:40:03.607656 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.100.1:42269
--local_ip_for_outbound_sockets=127.13.100.1
--tserver_master_addrs=127.13.100.62:35827
--webserver_port=38953
--webserver_interface=127.13.100.1
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.1.keytab
--principal=kudu/127.13.100.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.100.20:43917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 18:40:03.607707 28065 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc5af9cdcf5d4b16b729c8ac3ceb361c: Bootstrap starting.
I20260929 18:40:03.624600 28065 log.cc:824] T 00000000000000000000000000000000 P bc5af9cdcf5d4b16b729c8ac3ceb361c: Log is configured to *not* fsync() on all Append() calls
I20260929 18:40:03.633200 28065 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc5af9cdcf5d4b16b729c8ac3ceb361c: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 18:40:03.633594 28065 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc5af9cdcf5d4b16b729c8ac3ceb361c: Bootstrap complete.
I20260929 18:40:03.637743 28065 raft_consensus.cc:348] T 00000000000000000000000000000000 P bc5af9cdcf5d4b16b729c8ac3ceb361c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc5af9cdcf5d4b16b729c8ac3ceb361c" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 35827 } }
I20260929 18:40:03.638271 28065 raft_consensus.cc:729] T 00000000000000000000000000000000 P bc5af9cdcf5d4b16b729c8ac3ceb361c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc5af9cdcf5d4b16b729c8ac3ceb361c, State: Initialized, Role: FOLLOWER
I20260929 18:40:03.638758 28065 consensus_queue.cc:260] T 00000000000000000000000000000000 P bc5af9cdcf5d4b16b729c8ac3ceb361c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc5af9cdcf5d4b16b729c8ac3ceb361c" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 35827 } }
I20260929 18:40:03.638921 28065 raft_consensus.cc:388] T 00000000000000000000000000000000 P bc5af9cdcf5d4b16b729c8ac3ceb361c [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:40:03.639030 28065 raft_consensus.cc:482] T 00000000000000000000000000000000 P bc5af9cdcf5d4b16b729c8ac3ceb361c [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:40:03.639150 28065 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bc5af9cdcf5d4b16b729c8ac3ceb361c [term 1 FOLLOWER]: Advancing to term 2
I20260929 18:40:03.641153 28065 raft_consensus.cc:504] T 00000000000000000000000000000000 P bc5af9cdcf5d4b16b729c8ac3ceb361c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc5af9cdcf5d4b16b729c8ac3ceb361c" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 35827 } }
I20260929 18:40:03.641494 28065 leader_election.cc:302] T 00000000000000000000000000000000 P bc5af9cdcf5d4b16b729c8ac3ceb361c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bc5af9cdcf5d4b16b729c8ac3ceb361c; no voters:
I20260929 18:40:03.641834 28065 leader_election.cc:288] T 00000000000000000000000000000000 P bc5af9cdcf5d4b16b729c8ac3ceb361c [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 18:40:03.641995 28069 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bc5af9cdcf5d4b16b729c8ac3ceb361c [term 2 FOLLOWER]: Leader election won for term 2
I20260929 18:40:03.642269 28069 raft_consensus.cc:686] T 00000000000000000000000000000000 P bc5af9cdcf5d4b16b729c8ac3ceb361c [term 2 LEADER]: Becoming Leader. State: Replica: bc5af9cdcf5d4b16b729c8ac3ceb361c, State: Running, Role: LEADER
I20260929 18:40:03.642596 28069 consensus_queue.cc:237] T 00000000000000000000000000000000 P bc5af9cdcf5d4b16b729c8ac3ceb361c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc5af9cdcf5d4b16b729c8ac3ceb361c" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 35827 } }
I20260929 18:40:03.643110 28065 sys_catalog.cc:564] T 00000000000000000000000000000000 P bc5af9cdcf5d4b16b729c8ac3ceb361c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:40:03.643615 28069 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc5af9cdcf5d4b16b729c8ac3ceb361c [sys.catalog]: SysCatalogTable state changed. Reason: New leader bc5af9cdcf5d4b16b729c8ac3ceb361c. Latest consensus state: current_term: 2 leader_uuid: "bc5af9cdcf5d4b16b729c8ac3ceb361c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc5af9cdcf5d4b16b729c8ac3ceb361c" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 35827 } } }
I20260929 18:40:03.643744 28069 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc5af9cdcf5d4b16b729c8ac3ceb361c [sys.catalog]: This master's current role is: LEADER
I20260929 18:40:03.644377 28070 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc5af9cdcf5d4b16b729c8ac3ceb361c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "bc5af9cdcf5d4b16b729c8ac3ceb361c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc5af9cdcf5d4b16b729c8ac3ceb361c" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 35827 } } }
I20260929 18:40:03.644475 28070 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc5af9cdcf5d4b16b729c8ac3ceb361c [sys.catalog]: This master's current role is: LEADER
I20260929 18:40:03.645249 28073 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:40:03.650104 28073 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:40:03.650692 28073 catalog_manager.cc:1259] Loaded cluster ID: b85dd72d97cd477f92a9a3802549ebfa
I20260929 18:40:03.650753 28073 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:40:03.652099 28073 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260929 18:40:03.656955 28073 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260929 18:40:03.663000 28073 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:40:03.663918 28073 catalog_manager.cc:5879] T 00000000000000000000000000000000 P bc5af9cdcf5d4b16b729c8ac3ceb361c: Loaded TSK: 0
I20260929 18:40:03.665191 28073 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 18:40:03.733573 28067 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:40:03.733861 28067 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:40:03.733989 28067 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:40:03.738230 28067 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:40:03.738333 28067 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:40:03.738428 28067 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.100.1
I20260929 18:40:03.744253 28067 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.100.20:43917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.1.keytab
--principal=kudu/127.13.100.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.100.1:42269
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.100.1
--webserver_port=38953
--tserver_master_addrs=127.13.100.62:35827
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28067
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.100.1
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:40:03.745472 28067 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:40:03.746554 28067 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:40:03.756506 28095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:40:03.756474 28092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:40:03.756765 28093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:40:03.756933 28067 server_base.cc:1034] running on GCE node
I20260929 18:40:03.757347 28067 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:40:03.758105 28067 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:40:03.759290 28067 hybrid_clock.cc:648] HybridClock initialized: now 1790707203759250 us; error 55 us; skew 500 ppm
Sep 29 18:40:03 dist-test-slave-j1xs krb5kdc[27447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707203, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:40:03.762171 28067 init.cc:377] Logged in from keytab as kudu/127.13.100.1@KRBTEST.COM (short username kudu)
I20260929 18:40:03.763185 28067 webserver.cc:460] Webserver started at http://127.13.100.1:38953/ using document root <none> and password file <none>
I20260929 18:40:03.763748 28067 fs_manager.cc:359] Metadata directory not provided
I20260929 18:40:03.763824 28067 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:40:03.767872 28067 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 18:40:03.770179 28102 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:40:03.771546 28067 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 18:40:03.771651 28067 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/ts-0/wal
uuid: "9fe6f60bc87f4337a740ea25d9265b13"
format_stamp: "Formatted at 2026-09-29 18:40:02 on dist-test-slave-j1xs"
server_key: "be8aa85caa3b213daed6b404d4b10721"
server_key_iv: "85d7fe3f3ffe0c297cd2dbbe62149376"
server_key_version: "encryptionkey@0"
I20260929 18:40:03.772222 28067 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:40:03.793314 28067 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:40:03.796911 28067 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:40:03.797137 28067 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:40:03.797770 28067 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:40:03.798933 28109 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260929 18:40:03.803073 28067 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 18:40:03.803144 28067 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260929 18:40:03.803224 28067 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 18:40:03.805038 28067 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 18:40:03.805085 28067 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260929 18:40:03.805411 28109 tablet_bootstrap.cc:492] T 1338ed01901440118fbcca2a287b7ff8 P 9fe6f60bc87f4337a740ea25d9265b13: Bootstrap starting.
I20260929 18:40:03.815570 28067 rpc_server.cc:307] RPC server started. Bound to: 127.13.100.1:42269
I20260929 18:40:03.815697 28216 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.100.1:42269 every 8 connection(s)
I20260929 18:40:03.816562 28067 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/ts-0/data/info.pb
I20260929 18:40:03.823318 28109 log.cc:824] T 1338ed01901440118fbcca2a287b7ff8 P 9fe6f60bc87f4337a740ea25d9265b13: Log is configured to *not* fsync() on all Append() calls
I20260929 18:40:03.825335 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 28067
I20260929 18:40:03.825947 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.100.2:44645
--local_ip_for_outbound_sockets=127.13.100.2
--tserver_master_addrs=127.13.100.62:35827
--webserver_port=46185
--webserver_interface=127.13.100.2
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.2.keytab
--principal=kudu/127.13.100.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.100.20:43917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 18:40:03 dist-test-slave-j1xs krb5kdc[27447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707203, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.1@KRBTEST.COM for kudu/127.13.100.62@KRBTEST.COM
I20260929 18:40:03.831321 28109 tablet_bootstrap.cc:492] T 1338ed01901440118fbcca2a287b7ff8 P 9fe6f60bc87f4337a740ea25d9265b13: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 18:40:03.831787 28109 tablet_bootstrap.cc:492] T 1338ed01901440118fbcca2a287b7ff8 P 9fe6f60bc87f4337a740ea25d9265b13: Bootstrap complete.
I20260929 18:40:03.833040 28109 ts_tablet_manager.cc:1397] T 1338ed01901440118fbcca2a287b7ff8 P 9fe6f60bc87f4337a740ea25d9265b13: Time spent bootstrapping tablet: real 0.028s user 0.021s sys 0.004s
I20260929 18:40:03.834321 28220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:03.818685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:35827 (local address 127.13.100.1:51631)
0929 18:40:03.819226 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0929 18:40:03.819267 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 18:40:03.820067 (+ 800us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:40:03.821106 (+ 1039us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:40:03.821114 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:40:03.821524 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:40:03.822026 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:40:03.822042 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:40:03.823300 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:40:03.823303 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:40:03.824347 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:40:03.824358 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:40:03.825156 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:40:03.825160 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:40:03.825353 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:40:03.826093 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:40:03.826112 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:40:03.827998 (+ 1886us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:40:03.830679 (+ 2681us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:40:03.830685 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:40:03.830697 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:40:03.831009 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:40:03.831505 (+ 496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:40:03.831510 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:40:03.831511 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:40:03.831734 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:40:03.832523 (+ 789us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:40:03.832531 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:40:03.832817 (+ 286us) client_negotiation.cc:769] Sending connection context
0929 18:40:03.833018 (+ 201us) client_negotiation.cc:240] Negotiation successful
0929 18:40:03.833357 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":177,"threads_started":1}
I20260929 18:40:03.834530 28219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:03.818638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.1:51631 (local address 127.13.100.62:35827)
0929 18:40:03.818981 (+ 343us) server_negotiation.cc:206] Beginning negotiation
0929 18:40:03.819013 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0929 18:40:03.819385 (+ 372us) server_negotiation.cc:407] Connection header received
0929 18:40:03.820452 (+ 1067us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:40:03.820462 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:40:03.820737 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:40:03.821051 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:40:03.822185 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:03.823134 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:03.824504 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:03.824978 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:03.825182 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:40:03.828206 (+ 3024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:40:03.828211 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:40:03.828223 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:40:03.828250 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:40:03.830536 (+ 2286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:40:03.831135 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:40:03.831145 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:40:03.831153 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:40:03.831334 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:40:03.831848 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:40:03.831855 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:40:03.831858 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:40:03.832386 (+ 528us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:40:03.832548 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0929 18:40:03.833174 (+ 626us) server_negotiation.cc:299] Negotiation successful
0929 18:40:03.833530 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":112,"threads_started":1}
I20260929 18:40:03.835975 28109 raft_consensus.cc:348] T 1338ed01901440118fbcca2a287b7ff8 P 9fe6f60bc87f4337a740ea25d9265b13 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5aa041fdd8a4573b36ee7aee2085658" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 45395 } } peers { permanent_uuid: "2b4519ad4d74462ebfa5eba155184701" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 44645 } } peers { permanent_uuid: "9fe6f60bc87f4337a740ea25d9265b13" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 42269 } }
I20260929 18:40:03.836512 28109 raft_consensus.cc:729] T 1338ed01901440118fbcca2a287b7ff8 P 9fe6f60bc87f4337a740ea25d9265b13 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fe6f60bc87f4337a740ea25d9265b13, State: Initialized, Role: FOLLOWER
I20260929 18:40:03.836978 28109 consensus_queue.cc:260] T 1338ed01901440118fbcca2a287b7ff8 P 9fe6f60bc87f4337a740ea25d9265b13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5aa041fdd8a4573b36ee7aee2085658" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 45395 } } peers { permanent_uuid: "2b4519ad4d74462ebfa5eba155184701" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 44645 } } peers { permanent_uuid: "9fe6f60bc87f4337a740ea25d9265b13" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 42269 } }
I20260929 18:40:03.837157 28217 heartbeater.cc:347] Connected to a master server at 127.13.100.62:35827
I20260929 18:40:03.837430 28217 heartbeater.cc:464] Registering TS with master...
I20260929 18:40:03.837940 28109 ts_tablet_manager.cc:1428] T 1338ed01901440118fbcca2a287b7ff8 P 9fe6f60bc87f4337a740ea25d9265b13: Time spent starting tablet: real 0.005s user 0.008s sys 0.000s
I20260929 18:40:03.838085 28217 heartbeater.cc:511] Master 127.13.100.62:35827 requested a full tablet report, sending...
I20260929 18:40:03.840708 28029 ts_manager.cc:194] Registered new tserver with Master: 9fe6f60bc87f4337a740ea25d9265b13 (127.13.100.1:42269)
I20260929 18:40:03.844219 28029 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.100.1@KRBTEST.COM'} at 127.13.100.1:51631
I20260929 18:40:03.847136 28217 heartbeater.cc:503] Master 127.13.100.62:35827 was elected leader, sending a full tablet report...
Sep 29 18:40:03 dist-test-slave-j1xs krb5kdc[27447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707203, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.62@KRBTEST.COM for kudu/127.13.100.1@KRBTEST.COM
I20260929 18:40:03.854411 28224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:03.843470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.1:42269 (local address 127.0.0.1:48272)
0929 18:40:03.844219 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0929 18:40:03.844282 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0929 18:40:03.844469 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:40:03.845204 (+ 735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:40:03.845210 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:40:03.845619 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:40:03.845722 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:40:03.845729 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:40:03.846715 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:40:03.846720 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:40:03.847365 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:40:03.847375 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:40:03.848058 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:40:03.848062 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:40:03.848186 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:40:03.848856 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:40:03.848873 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:40:03.850458 (+ 1585us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:40:03.852682 (+ 2224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:40:03.852690 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:40:03.852694 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:40:03.852913 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:40:03.853228 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:40:03.853230 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:40:03.853231 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:40:03.853305 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:40:03.853824 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:40:03.853829 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:40:03.853988 (+ 159us) client_negotiation.cc:769] Sending connection context
0929 18:40:03.854121 (+ 133us) client_negotiation.cc:240] Negotiation successful
0929 18:40:03.854276 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":124,"threads_started":1}
I20260929 18:40:03.854844 28225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:03.843558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48272 (local address 127.13.100.1:42269)
0929 18:40:03.843927 (+ 369us) server_negotiation.cc:206] Beginning negotiation
0929 18:40:03.843937 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 18:40:03.844369 (+ 432us) server_negotiation.cc:407] Connection header received
0929 18:40:03.844656 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:40:03.844661 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:40:03.844861 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:40:03.844971 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:40:03.845839 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:03.846603 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:03.847478 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:03.847933 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:03.848051 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:40:03.850602 (+ 2551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:40:03.850613 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:40:03.850627 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:40:03.850654 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:40:03.852569 (+ 1915us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:40:03.853024 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:40:03.853030 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:40:03.853032 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:40:03.853134 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:40:03.853388 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:40:03.853392 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:40:03.853393 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:40:03.853697 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:40:03.853843 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0929 18:40:03.854232 (+ 389us) server_negotiation.cc:299] Negotiation successful
0929 18:40:03.854420 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":106,"threads_started":1}
I20260929 18:40:03.855819 28151 tablet_service.cc:1514] Processing DeleteTablet for tablet 1338ed01901440118fbcca2a287b7ff8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:40:03 UTC) from {username='kudu', principal='kudu/127.13.100.62@KRBTEST.COM'} at 127.0.0.1:48272
I20260929 18:40:03.856666 28226 tablet_replica.cc:330] stopping tablet replica
I20260929 18:40:03.856956 28226 raft_consensus.cc:2229] T 1338ed01901440118fbcca2a287b7ff8 P 9fe6f60bc87f4337a740ea25d9265b13 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:40:03.857208 28226 raft_consensus.cc:2258] T 1338ed01901440118fbcca2a287b7ff8 P 9fe6f60bc87f4337a740ea25d9265b13 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:40:03.858301 28226 ts_tablet_manager.cc:1905] T 1338ed01901440118fbcca2a287b7ff8 P 9fe6f60bc87f4337a740ea25d9265b13: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:40:03.862073 28226 ts_tablet_manager.cc:1918] T 1338ed01901440118fbcca2a287b7ff8 P 9fe6f60bc87f4337a740ea25d9265b13: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:40:03.862185 28226 log.cc:1196] T 1338ed01901440118fbcca2a287b7ff8 P 9fe6f60bc87f4337a740ea25d9265b13: Deleting WAL directory at /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/ts-0/wal/wals/1338ed01901440118fbcca2a287b7ff8
I20260929 18:40:03.862458 28226 ts_tablet_manager.cc:1939] T 1338ed01901440118fbcca2a287b7ff8 P 9fe6f60bc87f4337a740ea25d9265b13: Deleting consensus metadata
I20260929 18:40:03.864071 28013 catalog_manager.cc:4841] TS 9fe6f60bc87f4337a740ea25d9265b13 (127.13.100.1:42269): tablet 1338ed01901440118fbcca2a287b7ff8 (table test-table [id=00fdaf3ea6e74809935ca13bf796543c]) successfully deleted
W20260929 18:40:03.944553 28222 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:40:03.944795 28222 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:40:03.944837 28222 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:40:03.948915 28222 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:40:03.948972 28222 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:40:03.949060 28222 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.100.2
I20260929 18:40:03.954486 28222 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.100.20:43917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.2.keytab
--principal=kudu/127.13.100.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.100.2:44645
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.100.2
--webserver_port=46185
--tserver_master_addrs=127.13.100.62:35827
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28222
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.100.2
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:40:03.955461 28222 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:40:03.956434 28222 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:40:03.966276 28234 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:40:03.966279 28231 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:40:03.966300 28232 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:40:03.966744 28222 server_base.cc:1034] running on GCE node
I20260929 18:40:03.967145 28222 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:40:03.967720 28222 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:40:03.968885 28222 hybrid_clock.cc:648] HybridClock initialized: now 1790707203968859 us; error 39 us; skew 500 ppm
Sep 29 18:40:03 dist-test-slave-j1xs krb5kdc[27447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707203, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:40:03.971717 28222 init.cc:377] Logged in from keytab as kudu/127.13.100.2@KRBTEST.COM (short username kudu)
I20260929 18:40:03.972651 28222 webserver.cc:460] Webserver started at http://127.13.100.2:46185/ using document root <none> and password file <none>
I20260929 18:40:03.973256 28222 fs_manager.cc:359] Metadata directory not provided
I20260929 18:40:03.973353 28222 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:40:03.977373 28222 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 18:40:03.979583 28241 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:40:03.980856 28222 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 18:40:03.980994 28222 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/ts-1/wal
uuid: "2b4519ad4d74462ebfa5eba155184701"
format_stamp: "Formatted at 2026-09-29 18:40:02 on dist-test-slave-j1xs"
server_key: "bce4f1c80ce470c76043a3749f9a2130"
server_key_iv: "c3668ad0ef7af0f4ace653617c9fb506"
server_key_version: "encryptionkey@0"
I20260929 18:40:03.981731 28222 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:40:04.010775 28222 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:40:04.013727 28222 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:40:04.014003 28222 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:40:04.014626 28222 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:40:04.015476 28222 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:40:04.015528 28222 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:40:04.015599 28222 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:40:04.015638 28222 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:40:04.025077 28222 rpc_server.cc:307] RPC server started. Bound to: 127.13.100.2:44645
I20260929 18:40:04.025107 28354 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.100.2:44645 every 8 connection(s)
I20260929 18:40:04.026113 28222 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/ts-1/data/info.pb
I20260929 18:40:04.034139 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 28222
I20260929 18:40:04.034658 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.100.3:45395
--local_ip_for_outbound_sockets=127.13.100.3
--tserver_master_addrs=127.13.100.62:35827
--webserver_port=36755
--webserver_interface=127.13.100.3
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.3.keytab
--principal=kudu/127.13.100.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.100.20:43917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 18:40:04 dist-test-slave-j1xs krb5kdc[27447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707203, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.2@KRBTEST.COM for kudu/127.13.100.62@KRBTEST.COM
I20260929 18:40:04.040441 28219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:04.027931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.2:47305 (local address 127.13.100.62:35827)
0929 18:40:04.028084 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0929 18:40:04.028088 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:40:04.028881 (+ 793us) server_negotiation.cc:407] Connection header received
0929 18:40:04.030263 (+ 1382us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:40:04.030267 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:40:04.030321 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:40:04.030421 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:40:04.031556 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:04.032073 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:04.033180 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:04.033407 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:04.033515 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:40:04.036258 (+ 2743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:40:04.036263 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:40:04.036267 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:40:04.036300 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:40:04.038044 (+ 1744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:40:04.038601 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:40:04.038607 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:40:04.038612 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:40:04.038710 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:40:04.039088 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:40:04.039093 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:40:04.039096 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:40:04.039358 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:40:04.039470 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 18:40:04.040030 (+ 560us) server_negotiation.cc:299] Negotiation successful
0929 18:40:04.040190 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 18:40:04.041090 28357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:04.028294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:35827 (local address 127.13.100.2:47305)
0929 18:40:04.028747 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0929 18:40:04.028788 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 18:40:04.030068 (+ 1280us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:40:04.030587 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:40:04.030593 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:40:04.030942 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:40:04.031405 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:40:04.031418 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:40:04.032196 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:40:04.032199 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:40:04.033058 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:40:04.033067 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:40:04.033552 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:40:04.033556 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:40:04.033722 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:40:04.034409 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:40:04.034433 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:40:04.036088 (+ 1655us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:40:04.038171 (+ 2083us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:40:04.038176 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:40:04.038187 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:40:04.038486 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:40:04.038833 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:40:04.038837 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:40:04.038839 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:40:04.038984 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:40:04.039483 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:40:04.039488 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:40:04.039769 (+ 281us) client_negotiation.cc:769] Sending connection context
0929 18:40:04.039954 (+ 185us) client_negotiation.cc:240] Negotiation successful
0929 18:40:04.040216 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":106,"threads_started":1}
I20260929 18:40:04.042307 28355 heartbeater.cc:347] Connected to a master server at 127.13.100.62:35827
I20260929 18:40:04.042536 28355 heartbeater.cc:464] Registering TS with master...
I20260929 18:40:04.043026 28355 heartbeater.cc:511] Master 127.13.100.62:35827 requested a full tablet report, sending...
I20260929 18:40:04.044092 28029 ts_manager.cc:194] Registered new tserver with Master: 2b4519ad4d74462ebfa5eba155184701 (127.13.100.2:44645)
I20260929 18:40:04.044720 28029 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.100.2@KRBTEST.COM'} at 127.13.100.2:47305
W20260929 18:40:04.150223 28358 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:40:04.150478 28358 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:40:04.150522 28358 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:40:04.154503 28358 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:40:04.154557 28358 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:40:04.154615 28358 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.100.3
I20260929 18:40:04.160696 28358 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.100.20:43917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.3.keytab
--principal=kudu/127.13.100.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.100.3:45395
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.100.3
--webserver_port=36755
--tserver_master_addrs=127.13.100.62:35827
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28358
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.100.3
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:40:04.161736 28358 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:40:04.162878 28358 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:40:04.172803 28363 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:40:04.172818 28364 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:40:04.172930 28358 server_base.cc:1034] running on GCE node
W20260929 18:40:04.172818 28366 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:40:04.173555 28358 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:40:04.174355 28358 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:40:04.175529 28358 hybrid_clock.cc:648] HybridClock initialized: now 1790707204175514 us; error 27 us; skew 500 ppm
Sep 29 18:40:04 dist-test-slave-j1xs krb5kdc[27447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707204, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:40:04.178519 28358 init.cc:377] Logged in from keytab as kudu/127.13.100.3@KRBTEST.COM (short username kudu)
I20260929 18:40:04.179394 28358 webserver.cc:460] Webserver started at http://127.13.100.3:36755/ using document root <none> and password file <none>
I20260929 18:40:04.179975 28358 fs_manager.cc:359] Metadata directory not provided
I20260929 18:40:04.180074 28358 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:40:04.183902 28358 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 18:40:04.186110 28373 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:40:04.187309 28358 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:40:04.187435 28358 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/ts-2/wal
uuid: "a5aa041fdd8a4573b36ee7aee2085658"
format_stamp: "Formatted at 2026-09-29 18:40:02 on dist-test-slave-j1xs"
server_key: "351f8f834e1c4853a9ef0f872f19da00"
server_key_iv: "8a1dbdae6e6a05b37f329313c66820da"
server_key_version: "encryptionkey@0"
I20260929 18:40:04.187973 28358 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:40:04.212128 28358 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:40:04.215409 28358 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:40:04.215606 28358 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:40:04.216229 28358 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:40:04.217034 28358 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:40:04.217101 28358 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:40:04.217170 28358 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:40:04.217209 28358 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:40:04.225725 28358 rpc_server.cc:307] RPC server started. Bound to: 127.13.100.3:45395
I20260929 18:40:04.225796 28486 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.100.3:45395 every 8 connection(s)
I20260929 18:40:04.226713 28358 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/ts-2/data/info.pb
I20260929 18:40:04.231980 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 28358
Sep 29 18:40:04 dist-test-slave-j1xs krb5kdc[27447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707204, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.3@KRBTEST.COM for kudu/127.13.100.62@KRBTEST.COM
I20260929 18:40:04.241698 28490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:04.232613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55692 (local address 127.13.100.62:35827)
0929 18:40:04.232853 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0929 18:40:04.232857 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:40:04.233072 (+ 215us) server_negotiation.cc:407] Connection header received
0929 18:40:04.233195 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:40:04.233197 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:40:04.233234 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:40:04.233305 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:40:04.234187 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:04.234894 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:04.236008 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:04.236236 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:04.236310 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:40:04.237813 (+ 1503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:40:04.237816 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:40:04.237817 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:40:04.237847 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:40:04.239789 (+ 1942us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:40:04.240329 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:40:04.240332 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:40:04.240334 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:40:04.240440 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:40:04.240771 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:40:04.240773 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:40:04.240774 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:40:04.241002 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:40:04.241198 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0929 18:40:04.241417 (+ 219us) server_negotiation.cc:299] Negotiation successful
0929 18:40:04.241528 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":160,"thread_start_us":77,"threads_started":1}
I20260929 18:40:04.242141 28219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:04.228476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.3:41177 (local address 127.13.100.62:35827)
0929 18:40:04.228649 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0929 18:40:04.228653 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:40:04.229412 (+ 759us) server_negotiation.cc:407] Connection header received
0929 18:40:04.230787 (+ 1375us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:40:04.230790 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:40:04.230841 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:40:04.230928 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:40:04.232030 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:04.232728 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:04.234410 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:04.234776 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:04.234872 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:40:04.237535 (+ 2663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:40:04.237543 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:40:04.237546 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:40:04.237568 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:40:04.239754 (+ 2186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:40:04.240405 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:40:04.240409 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:40:04.240410 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:40:04.240477 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:40:04.240874 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:40:04.240877 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:40:04.240879 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:40:04.241043 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:40:04.241122 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 18:40:04.241774 (+ 652us) server_negotiation.cc:299] Negotiation successful
0929 18:40:04.241950 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":29,"server-negotiator.queue_time_us":76}
I20260929 18:40:04.243750 28489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:04.228835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:35827 (local address 127.13.100.3:41177)
0929 18:40:04.229266 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0929 18:40:04.229306 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 18:40:04.230570 (+ 1264us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:40:04.231078 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:40:04.231084 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:40:04.231410 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:40:04.231878 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:40:04.231891 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:40:04.232867 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:40:04.232870 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:40:04.234305 (+ 1435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:40:04.234314 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:40:04.234907 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:40:04.234909 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:40:04.235115 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:40:04.235714 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:40:04.235733 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:40:04.237377 (+ 1644us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:40:04.239918 (+ 2541us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:40:04.239924 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:40:04.239940 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:40:04.240313 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:40:04.240568 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:40:04.240571 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:40:04.240573 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:40:04.240767 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:40:04.241139 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:40:04.241145 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:40:04.241460 (+ 315us) client_negotiation.cc:769] Sending connection context
0929 18:40:04.242301 (+ 841us) client_negotiation.cc:240] Negotiation successful
0929 18:40:04.242646 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":114,"threads_started":1}
I20260929 18:40:04.245373 28487 heartbeater.cc:347] Connected to a master server at 127.13.100.62:35827
I20260929 18:40:04.245638 28487 heartbeater.cc:464] Registering TS with master...
I20260929 18:40:04.246196 28487 heartbeater.cc:511] Master 127.13.100.62:35827 requested a full tablet report, sending...
I20260929 18:40:04.247181 28029 ts_manager.cc:194] Registered new tserver with Master: a5aa041fdd8a4573b36ee7aee2085658 (127.13.100.3:45395)
I20260929 18:40:04.247938 28029 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.100.3@KRBTEST.COM'} at 127.13.100.3:41177
I20260929 18:40:04.255368 13712 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 18:40:04.259774 28219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:04.256016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55700 (local address 127.13.100.62:35827)
0929 18:40:04.256177 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0929 18:40:04.256181 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:40:04.256356 (+ 175us) server_negotiation.cc:407] Connection header received
0929 18:40:04.256553 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:40:04.256556 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:40:04.256668 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:40:04.256761 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:40:04.256982 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:04.257493 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:04.258366 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:04.258607 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:04.258707 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:40:04.259027 (+ 320us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:40:04.259414 (+ 387us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:40:04.259525 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 18:40:04.259556 (+ 31us) server_negotiation.cc:299] Negotiation successful
0929 18:40:04.259603 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 18:40:04.260437 28029 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:55700:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:40:04.263972 28029 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:40:04.272086 28151 tablet_service.cc:1467] Processing CreateTablet for tablet 83f68cf6db45459499932b2fcdd186d6 (DEFAULT_TABLE table=test-table [id=386248b7ea6d44a6bc7fe242d1d84848]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:40:04.272738 28151 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83f68cf6db45459499932b2fcdd186d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:40:04.275601 28109 tablet_bootstrap.cc:492] T 83f68cf6db45459499932b2fcdd186d6 P 9fe6f60bc87f4337a740ea25d9265b13: Bootstrap starting.
I20260929 18:40:04.276547 28109 tablet_bootstrap.cc:654] T 83f68cf6db45459499932b2fcdd186d6 P 9fe6f60bc87f4337a740ea25d9265b13: Neither blocks nor log segments found. Creating new log.
I20260929 18:40:04.276754 28224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:04.270942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.2:44645 (local address 127.0.0.1:36426)
0929 18:40:04.271107 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0929 18:40:04.271123 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 18:40:04.271198 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:40:04.272182 (+ 984us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:40:04.272185 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:40:04.272202 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:40:04.272329 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:40:04.272336 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:40:04.273713 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:40:04.273719 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:40:04.275146 (+ 1427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:40:04.275156 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:40:04.276295 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:40:04.276298 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:40:04.276402 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:40:04.276447 (+ 45us) client_negotiation.cc:769] Sending connection context
0929 18:40:04.276529 (+ 82us) client_negotiation.cc:240] Negotiation successful
0929 18:40:04.276620 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86}
I20260929 18:40:04.276839 28496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:04.271188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36426 (local address 127.13.100.2:44645)
0929 18:40:04.271678 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0929 18:40:04.271691 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 18:40:04.271714 (+ 23us) server_negotiation.cc:407] Connection header received
0929 18:40:04.271797 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:40:04.271806 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:40:04.272035 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:40:04.272204 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:40:04.272441 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:04.273563 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:04.275305 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:04.276177 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:04.276292 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:40:04.276439 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0929 18:40:04.276570 (+ 131us) server_negotiation.cc:299] Negotiation successful
0929 18:40:04.276668 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":157,"threads_started":1}
I20260929 18:40:04.277514 28497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:04.271259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.3:45395 (local address 127.0.0.1:50434)
0929 18:40:04.271549 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0929 18:40:04.271563 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:40:04.271642 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:40:04.272519 (+ 877us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:40:04.272522 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:40:04.272540 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:40:04.272617 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:40:04.272623 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:40:04.274223 (+ 1600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:40:04.274226 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:40:04.275372 (+ 1146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:40:04.275380 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:40:04.277139 (+ 1759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:40:04.277142 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:40:04.277234 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:40:04.277256 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 18:40:04.277302 (+ 46us) client_negotiation.cc:240] Negotiation successful
0929 18:40:04.277387 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":108,"threads_started":1}
I20260929 18:40:04.278297 28498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:04.271711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50434 (local address 127.13.100.3:45395)
0929 18:40:04.272039 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0929 18:40:04.272059 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 18:40:04.272081 (+ 22us) server_negotiation.cc:407] Connection header received
0929 18:40:04.272155 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:40:04.272159 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:40:04.272337 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:40:04.272524 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:40:04.272712 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:04.274105 (+ 1393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:04.275484 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:04.276828 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:04.277869 (+ 1041us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:40:04.278028 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0929 18:40:04.278076 (+ 48us) server_negotiation.cc:299] Negotiation successful
0929 18:40:04.278161 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230,"thread_start_us":122,"threads_started":1}
I20260929 18:40:04.278553 28109 tablet_bootstrap.cc:492] T 83f68cf6db45459499932b2fcdd186d6 P 9fe6f60bc87f4337a740ea25d9265b13: No bootstrap required, opened a new log
I20260929 18:40:04.278654 28109 ts_tablet_manager.cc:1397] T 83f68cf6db45459499932b2fcdd186d6 P 9fe6f60bc87f4337a740ea25d9265b13: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 18:40:04.279032 28109 raft_consensus.cc:348] T 83f68cf6db45459499932b2fcdd186d6 P 9fe6f60bc87f4337a740ea25d9265b13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b4519ad4d74462ebfa5eba155184701" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 44645 } } peers { permanent_uuid: "a5aa041fdd8a4573b36ee7aee2085658" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 45395 } } peers { permanent_uuid: "9fe6f60bc87f4337a740ea25d9265b13" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 42269 } }
I20260929 18:40:04.279158 28109 raft_consensus.cc:374] T 83f68cf6db45459499932b2fcdd186d6 P 9fe6f60bc87f4337a740ea25d9265b13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:40:04.279210 28109 raft_consensus.cc:729] T 83f68cf6db45459499932b2fcdd186d6 P 9fe6f60bc87f4337a740ea25d9265b13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fe6f60bc87f4337a740ea25d9265b13, State: Initialized, Role: FOLLOWER
I20260929 18:40:04.279333 28109 consensus_queue.cc:260] T 83f68cf6db45459499932b2fcdd186d6 P 9fe6f60bc87f4337a740ea25d9265b13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b4519ad4d74462ebfa5eba155184701" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 44645 } } peers { permanent_uuid: "a5aa041fdd8a4573b36ee7aee2085658" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 45395 } } peers { permanent_uuid: "9fe6f60bc87f4337a740ea25d9265b13" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 42269 } }
I20260929 18:40:04.279589 28109 ts_tablet_manager.cc:1428] T 83f68cf6db45459499932b2fcdd186d6 P 9fe6f60bc87f4337a740ea25d9265b13: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260929 18:40:04.280064 28289 tablet_service.cc:1467] Processing CreateTablet for tablet 83f68cf6db45459499932b2fcdd186d6 (DEFAULT_TABLE table=test-table [id=386248b7ea6d44a6bc7fe242d1d84848]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:40:04.281260 28289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83f68cf6db45459499932b2fcdd186d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:40:04.282549 28421 tablet_service.cc:1467] Processing CreateTablet for tablet 83f68cf6db45459499932b2fcdd186d6 (DEFAULT_TABLE table=test-table [id=386248b7ea6d44a6bc7fe242d1d84848]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:40:04.283614 28421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83f68cf6db45459499932b2fcdd186d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:40:04.288398 28500 tablet_bootstrap.cc:492] T 83f68cf6db45459499932b2fcdd186d6 P 2b4519ad4d74462ebfa5eba155184701: Bootstrap starting.
I20260929 18:40:04.290544 28501 tablet_bootstrap.cc:492] T 83f68cf6db45459499932b2fcdd186d6 P a5aa041fdd8a4573b36ee7aee2085658: Bootstrap starting.
I20260929 18:40:04.291232 28500 tablet_bootstrap.cc:654] T 83f68cf6db45459499932b2fcdd186d6 P 2b4519ad4d74462ebfa5eba155184701: Neither blocks nor log segments found. Creating new log.
I20260929 18:40:04.292155 28500 log.cc:824] T 83f68cf6db45459499932b2fcdd186d6 P 2b4519ad4d74462ebfa5eba155184701: Log is configured to *not* fsync() on all Append() calls
I20260929 18:40:04.293300 28501 tablet_bootstrap.cc:654] T 83f68cf6db45459499932b2fcdd186d6 P a5aa041fdd8a4573b36ee7aee2085658: Neither blocks nor log segments found. Creating new log.
I20260929 18:40:04.294152 28501 log.cc:824] T 83f68cf6db45459499932b2fcdd186d6 P a5aa041fdd8a4573b36ee7aee2085658: Log is configured to *not* fsync() on all Append() calls
I20260929 18:40:04.294464 28500 tablet_bootstrap.cc:492] T 83f68cf6db45459499932b2fcdd186d6 P 2b4519ad4d74462ebfa5eba155184701: No bootstrap required, opened a new log
I20260929 18:40:04.294675 28500 ts_tablet_manager.cc:1397] T 83f68cf6db45459499932b2fcdd186d6 P 2b4519ad4d74462ebfa5eba155184701: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260929 18:40:04.296306 28501 tablet_bootstrap.cc:492] T 83f68cf6db45459499932b2fcdd186d6 P a5aa041fdd8a4573b36ee7aee2085658: No bootstrap required, opened a new log
I20260929 18:40:04.296499 28501 ts_tablet_manager.cc:1397] T 83f68cf6db45459499932b2fcdd186d6 P a5aa041fdd8a4573b36ee7aee2085658: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 18:40:04.298658 28500 raft_consensus.cc:348] T 83f68cf6db45459499932b2fcdd186d6 P 2b4519ad4d74462ebfa5eba155184701 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b4519ad4d74462ebfa5eba155184701" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 44645 } } peers { permanent_uuid: "a5aa041fdd8a4573b36ee7aee2085658" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 45395 } } peers { permanent_uuid: "9fe6f60bc87f4337a740ea25d9265b13" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 42269 } }
I20260929 18:40:04.298941 28500 raft_consensus.cc:374] T 83f68cf6db45459499932b2fcdd186d6 P 2b4519ad4d74462ebfa5eba155184701 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:40:04.298991 28500 raft_consensus.cc:729] T 83f68cf6db45459499932b2fcdd186d6 P 2b4519ad4d74462ebfa5eba155184701 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b4519ad4d74462ebfa5eba155184701, State: Initialized, Role: FOLLOWER
I20260929 18:40:04.299461 28500 consensus_queue.cc:260] T 83f68cf6db45459499932b2fcdd186d6 P 2b4519ad4d74462ebfa5eba155184701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b4519ad4d74462ebfa5eba155184701" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 44645 } } peers { permanent_uuid: "a5aa041fdd8a4573b36ee7aee2085658" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 45395 } } peers { permanent_uuid: "9fe6f60bc87f4337a740ea25d9265b13" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 42269 } }
I20260929 18:40:04.300175 28355 heartbeater.cc:503] Master 127.13.100.62:35827 was elected leader, sending a full tablet report...
I20260929 18:40:04.300297 28501 raft_consensus.cc:348] T 83f68cf6db45459499932b2fcdd186d6 P a5aa041fdd8a4573b36ee7aee2085658 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b4519ad4d74462ebfa5eba155184701" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 44645 } } peers { permanent_uuid: "a5aa041fdd8a4573b36ee7aee2085658" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 45395 } } peers { permanent_uuid: "9fe6f60bc87f4337a740ea25d9265b13" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 42269 } }
I20260929 18:40:04.300542 28501 raft_consensus.cc:374] T 83f68cf6db45459499932b2fcdd186d6 P a5aa041fdd8a4573b36ee7aee2085658 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:40:04.300567 28500 ts_tablet_manager.cc:1428] T 83f68cf6db45459499932b2fcdd186d6 P 2b4519ad4d74462ebfa5eba155184701: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 18:40:04.300606 28501 raft_consensus.cc:729] T 83f68cf6db45459499932b2fcdd186d6 P a5aa041fdd8a4573b36ee7aee2085658 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5aa041fdd8a4573b36ee7aee2085658, State: Initialized, Role: FOLLOWER
I20260929 18:40:04.301016 28501 consensus_queue.cc:260] T 83f68cf6db45459499932b2fcdd186d6 P a5aa041fdd8a4573b36ee7aee2085658 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b4519ad4d74462ebfa5eba155184701" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 44645 } } peers { permanent_uuid: "a5aa041fdd8a4573b36ee7aee2085658" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 45395 } } peers { permanent_uuid: "9fe6f60bc87f4337a740ea25d9265b13" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 42269 } }
I20260929 18:40:04.301688 28487 heartbeater.cc:503] Master 127.13.100.62:35827 was elected leader, sending a full tablet report...
I20260929 18:40:04.302011 28501 ts_tablet_manager.cc:1428] T 83f68cf6db45459499932b2fcdd186d6 P a5aa041fdd8a4573b36ee7aee2085658: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
W20260929 18:40:04.317926 28218 tablet.cc:2369] T 83f68cf6db45459499932b2fcdd186d6 P 9fe6f60bc87f4337a740ea25d9265b13: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:40:04.367370 28505 raft_consensus.cc:482] T 83f68cf6db45459499932b2fcdd186d6 P a5aa041fdd8a4573b36ee7aee2085658 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:40:04.367612 28505 raft_consensus.cc:504] T 83f68cf6db45459499932b2fcdd186d6 P a5aa041fdd8a4573b36ee7aee2085658 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b4519ad4d74462ebfa5eba155184701" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 44645 } } peers { permanent_uuid: "a5aa041fdd8a4573b36ee7aee2085658" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 45395 } } peers { permanent_uuid: "9fe6f60bc87f4337a740ea25d9265b13" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 42269 } }
I20260929 18:40:04.369566 28505 leader_election.cc:288] T 83f68cf6db45459499932b2fcdd186d6 P a5aa041fdd8a4573b36ee7aee2085658 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2b4519ad4d74462ebfa5eba155184701 (127.13.100.2:44645), 9fe6f60bc87f4337a740ea25d9265b13 (127.13.100.1:42269)
I20260929 18:40:04.374495 28489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:04.369257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.2:44645 (local address 127.13.100.3:60585)
0929 18:40:04.369637 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0929 18:40:04.369652 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 18:40:04.369917 (+ 265us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:40:04.370367 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:40:04.370406 (+ 39us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:40:04.370422 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:40:04.370538 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:40:04.370548 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:40:04.371896 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:40:04.371899 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:40:04.373056 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:40:04.373067 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:40:04.373960 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:40:04.373963 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:40:04.374104 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:40:04.374163 (+ 59us) client_negotiation.cc:769] Sending connection context
0929 18:40:04.374225 (+ 62us) client_negotiation.cc:240] Negotiation successful
0929 18:40:04.374310 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"mutex_wait_us":7}
I20260929 18:40:04.375006 28496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:04.369401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.3:60585 (local address 127.13.100.2:44645)
0929 18:40:04.369537 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0929 18:40:04.369541 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:40:04.369770 (+ 229us) server_negotiation.cc:407] Connection header received
0929 18:40:04.370143 (+ 373us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:40:04.370146 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:40:04.370198 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:40:04.370263 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:40:04.370917 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:04.371765 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:04.373195 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:04.373780 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:04.374500 (+ 720us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:40:04.374604 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 18:40:04.374673 (+ 69us) server_negotiation.cc:299] Negotiation successful
0929 18:40:04.374825 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 18:40:04.375588 28309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83f68cf6db45459499932b2fcdd186d6" candidate_uuid: "a5aa041fdd8a4573b36ee7aee2085658" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b4519ad4d74462ebfa5eba155184701" is_pre_election: true
I20260929 18:40:04.376031 28309 raft_consensus.cc:2454] T 83f68cf6db45459499932b2fcdd186d6 P 2b4519ad4d74462ebfa5eba155184701 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5aa041fdd8a4573b36ee7aee2085658 in term 0.
I20260929 18:40:04.376514 28377 leader_election.cc:302] T 83f68cf6db45459499932b2fcdd186d6 P a5aa041fdd8a4573b36ee7aee2085658 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b4519ad4d74462ebfa5eba155184701, a5aa041fdd8a4573b36ee7aee2085658; no voters:
I20260929 18:40:04.376835 28505 raft_consensus.cc:2781] T 83f68cf6db45459499932b2fcdd186d6 P a5aa041fdd8a4573b36ee7aee2085658 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:40:04.376942 28505 raft_consensus.cc:482] T 83f68cf6db45459499932b2fcdd186d6 P a5aa041fdd8a4573b36ee7aee2085658 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:40:04.377003 28505 raft_consensus.cc:3037] T 83f68cf6db45459499932b2fcdd186d6 P a5aa041fdd8a4573b36ee7aee2085658 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:40:04.378268 28506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:04.369299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.1:42269 (local address 127.13.100.3:47773)
0929 18:40:04.369939 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0929 18:40:04.369953 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:40:04.370067 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:40:04.373959 (+ 3892us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:40:04.373962 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:40:04.373979 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:40:04.374132 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:40:04.374157 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:40:04.375230 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:40:04.375233 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:40:04.376658 (+ 1425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:40:04.376669 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:40:04.377739 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:40:04.377743 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:40:04.377871 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:40:04.377935 (+ 64us) client_negotiation.cc:769] Sending connection context
0929 18:40:04.378026 (+ 91us) client_negotiation.cc:240] Negotiation successful
0929 18:40:04.378092 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"mutex_wait_us":45,"thread_start_us":123,"threads_started":1}
I20260929 18:40:04.378420 28505 raft_consensus.cc:504] T 83f68cf6db45459499932b2fcdd186d6 P a5aa041fdd8a4573b36ee7aee2085658 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b4519ad4d74462ebfa5eba155184701" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 44645 } } peers { permanent_uuid: "a5aa041fdd8a4573b36ee7aee2085658" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 45395 } } peers { permanent_uuid: "9fe6f60bc87f4337a740ea25d9265b13" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 42269 } }
I20260929 18:40:04.378298 28507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:04.369532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.3:47773 (local address 127.13.100.1:42269)
0929 18:40:04.373555 (+ 4023us) server_negotiation.cc:206] Beginning negotiation
0929 18:40:04.373560 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:40:04.373575 (+ 15us) server_negotiation.cc:407] Connection header received
0929 18:40:04.373625 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:40:04.373627 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:40:04.373694 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:40:04.373809 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:40:04.374286 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:04.375130 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:04.376775 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:04.377586 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:04.377714 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:40:04.377849 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0929 18:40:04.378061 (+ 212us) server_negotiation.cc:299] Negotiation successful
0929 18:40:04.378145 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":61,"threads_started":1}
I20260929 18:40:04.378968 28505 leader_election.cc:288] T 83f68cf6db45459499932b2fcdd186d6 P a5aa041fdd8a4573b36ee7aee2085658 [CANDIDATE]: Term 1 election: Requested vote from peers 2b4519ad4d74462ebfa5eba155184701 (127.13.100.2:44645), 9fe6f60bc87f4337a740ea25d9265b13 (127.13.100.1:42269)
I20260929 18:40:04.379101 28309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83f68cf6db45459499932b2fcdd186d6" candidate_uuid: "a5aa041fdd8a4573b36ee7aee2085658" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b4519ad4d74462ebfa5eba155184701"
I20260929 18:40:04.379222 28309 raft_consensus.cc:3037] T 83f68cf6db45459499932b2fcdd186d6 P 2b4519ad4d74462ebfa5eba155184701 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:40:04.379580 28170 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83f68cf6db45459499932b2fcdd186d6" candidate_uuid: "a5aa041fdd8a4573b36ee7aee2085658" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9fe6f60bc87f4337a740ea25d9265b13"
I20260929 18:40:04.379853 28170 raft_consensus.cc:3037] T 83f68cf6db45459499932b2fcdd186d6 P 9fe6f60bc87f4337a740ea25d9265b13 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:40:04.380486 28309 raft_consensus.cc:2454] T 83f68cf6db45459499932b2fcdd186d6 P 2b4519ad4d74462ebfa5eba155184701 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5aa041fdd8a4573b36ee7aee2085658 in term 1.
I20260929 18:40:04.380774 28377 leader_election.cc:302] T 83f68cf6db45459499932b2fcdd186d6 P a5aa041fdd8a4573b36ee7aee2085658 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b4519ad4d74462ebfa5eba155184701, a5aa041fdd8a4573b36ee7aee2085658; no voters:
I20260929 18:40:04.381018 28170 raft_consensus.cc:2454] T 83f68cf6db45459499932b2fcdd186d6 P 9fe6f60bc87f4337a740ea25d9265b13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5aa041fdd8a4573b36ee7aee2085658 in term 1.
I20260929 18:40:04.379580 28171 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83f68cf6db45459499932b2fcdd186d6" candidate_uuid: "a5aa041fdd8a4573b36ee7aee2085658" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9fe6f60bc87f4337a740ea25d9265b13" is_pre_election: true
I20260929 18:40:04.381312 28171 raft_consensus.cc:2362] T 83f68cf6db45459499932b2fcdd186d6 P 9fe6f60bc87f4337a740ea25d9265b13 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate a5aa041fdd8a4573b36ee7aee2085658 in term 1. Re-sending same reply.
I20260929 18:40:04.381493 28505 raft_consensus.cc:2781] T 83f68cf6db45459499932b2fcdd186d6 P a5aa041fdd8a4573b36ee7aee2085658 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:40:04.381713 28505 raft_consensus.cc:686] T 83f68cf6db45459499932b2fcdd186d6 P a5aa041fdd8a4573b36ee7aee2085658 [term 1 LEADER]: Becoming Leader. State: Replica: a5aa041fdd8a4573b36ee7aee2085658, State: Running, Role: LEADER
I20260929 18:40:04.382082 28505 consensus_queue.cc:237] T 83f68cf6db45459499932b2fcdd186d6 P a5aa041fdd8a4573b36ee7aee2085658 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b4519ad4d74462ebfa5eba155184701" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 44645 } } peers { permanent_uuid: "a5aa041fdd8a4573b36ee7aee2085658" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 45395 } } peers { permanent_uuid: "9fe6f60bc87f4337a740ea25d9265b13" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 42269 } }
I20260929 18:40:04.385066 28029 catalog_manager.cc:5495] T 83f68cf6db45459499932b2fcdd186d6 P a5aa041fdd8a4573b36ee7aee2085658 reported cstate change: term changed from 0 to 1, leader changed from <none> to a5aa041fdd8a4573b36ee7aee2085658 (127.13.100.3). New cstate: current_term: 1 leader_uuid: "a5aa041fdd8a4573b36ee7aee2085658" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b4519ad4d74462ebfa5eba155184701" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 44645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5aa041fdd8a4573b36ee7aee2085658" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 45395 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9fe6f60bc87f4337a740ea25d9265b13" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 42269 } health_report { overall_health: UNKNOWN } } }
I20260929 18:40:04.401360 28498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:04.398025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50450 (local address 127.13.100.3:45395)
0929 18:40:04.398147 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0929 18:40:04.398150 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:40:04.398164 (+ 14us) server_negotiation.cc:407] Connection header received
0929 18:40:04.398295 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:40:04.398297 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:40:04.398359 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:40:04.398448 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:40:04.398680 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:04.399170 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:04.400005 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:04.400240 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:04.400327 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:40:04.400642 (+ 315us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:40:04.401031 (+ 389us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:40:04.401131 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 18:40:04.401204 (+ 73us) server_negotiation.cc:299] Negotiation successful
0929 18:40:04.401253 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 18:40:04.406908 28171 raft_consensus.cc:1261] T 83f68cf6db45459499932b2fcdd186d6 P 9fe6f60bc87f4337a740ea25d9265b13 [term 1 FOLLOWER]: Refusing update from remote peer a5aa041fdd8a4573b36ee7aee2085658: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:40:04.406911 28309 raft_consensus.cc:1261] T 83f68cf6db45459499932b2fcdd186d6 P 2b4519ad4d74462ebfa5eba155184701 [term 1 FOLLOWER]: Refusing update from remote peer a5aa041fdd8a4573b36ee7aee2085658: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:40:04.407706 28508 consensus_queue.cc:1035] T 83f68cf6db45459499932b2fcdd186d6 P a5aa041fdd8a4573b36ee7aee2085658 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b4519ad4d74462ebfa5eba155184701" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:40:04.407877 28505 consensus_queue.cc:1035] T 83f68cf6db45459499932b2fcdd186d6 P a5aa041fdd8a4573b36ee7aee2085658 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9fe6f60bc87f4337a740ea25d9265b13" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 42269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:40:04.422200 28027 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:55700:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:40:04.422385 28027 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:55700:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:40:04.424738 28027 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bc5af9cdcf5d4b16b729c8ac3ceb361c: Sending DeleteTablet for 3 replicas of tablet 83f68cf6db45459499932b2fcdd186d6
I20260929 18:40:04.425402 28421 tablet_service.cc:1514] Processing DeleteTablet for tablet 83f68cf6db45459499932b2fcdd186d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:40:04 UTC) from {username='kudu', principal='kudu/127.13.100.62@KRBTEST.COM'} at 127.0.0.1:50434
I20260929 18:40:04.425413 28289 tablet_service.cc:1514] Processing DeleteTablet for tablet 83f68cf6db45459499932b2fcdd186d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:40:04 UTC) from {username='kudu', principal='kudu/127.13.100.62@KRBTEST.COM'} at 127.0.0.1:36426
I20260929 18:40:04.425863 28521 tablet_replica.cc:330] T 83f68cf6db45459499932b2fcdd186d6 P a5aa041fdd8a4573b36ee7aee2085658: stopping tablet replica
I20260929 18:40:04.426144 28151 tablet_service.cc:1514] Processing DeleteTablet for tablet 83f68cf6db45459499932b2fcdd186d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:40:04 UTC) from {username='kudu', principal='kudu/127.13.100.62@KRBTEST.COM'} at 127.0.0.1:48272
I20260929 18:40:04.426239 28521 raft_consensus.cc:2229] T 83f68cf6db45459499932b2fcdd186d6 P a5aa041fdd8a4573b36ee7aee2085658 [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:40:04.426488 28523 tablet_replica.cc:330] T 83f68cf6db45459499932b2fcdd186d6 P 9fe6f60bc87f4337a740ea25d9265b13: stopping tablet replica
I20260929 18:40:04.426537 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 28067
I20260929 18:40:04.426640 28521 raft_consensus.cc:2258] T 83f68cf6db45459499932b2fcdd186d6 P a5aa041fdd8a4573b36ee7aee2085658 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:40:04.426743 28523 raft_consensus.cc:2229] T 83f68cf6db45459499932b2fcdd186d6 P 9fe6f60bc87f4337a740ea25d9265b13 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:40:04.426935 28523 raft_consensus.cc:2258] T 83f68cf6db45459499932b2fcdd186d6 P 9fe6f60bc87f4337a740ea25d9265b13 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:40:04.428170 28521 ts_tablet_manager.cc:1905] T 83f68cf6db45459499932b2fcdd186d6 P a5aa041fdd8a4573b36ee7aee2085658: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:40:04.430114 28522 tablet_replica.cc:330] T 83f68cf6db45459499932b2fcdd186d6 P 2b4519ad4d74462ebfa5eba155184701: stopping tablet replica
I20260929 18:40:04.430460 28522 raft_consensus.cc:2229] T 83f68cf6db45459499932b2fcdd186d6 P 2b4519ad4d74462ebfa5eba155184701 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:40:04.430840 28522 raft_consensus.cc:2258] T 83f68cf6db45459499932b2fcdd186d6 P 2b4519ad4d74462ebfa5eba155184701 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:40:04.431751 28521 ts_tablet_manager.cc:1918] T 83f68cf6db45459499932b2fcdd186d6 P a5aa041fdd8a4573b36ee7aee2085658: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:40:04.431864 28521 log.cc:1196] T 83f68cf6db45459499932b2fcdd186d6 P a5aa041fdd8a4573b36ee7aee2085658: Deleting WAL directory at /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/ts-2/wal/wals/83f68cf6db45459499932b2fcdd186d6
I20260929 18:40:04.432211 28521 ts_tablet_manager.cc:1939] T 83f68cf6db45459499932b2fcdd186d6 P a5aa041fdd8a4573b36ee7aee2085658: Deleting consensus metadata
I20260929 18:40:04.432593 28522 ts_tablet_manager.cc:1905] T 83f68cf6db45459499932b2fcdd186d6 P 2b4519ad4d74462ebfa5eba155184701: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:40:04.433465 28014 catalog_manager.cc:4841] TS a5aa041fdd8a4573b36ee7aee2085658 (127.13.100.3:45395): tablet 83f68cf6db45459499932b2fcdd186d6 (table test-table [id=386248b7ea6d44a6bc7fe242d1d84848]) successfully deleted
I20260929 18:40:04.435225 28522 ts_tablet_manager.cc:1918] T 83f68cf6db45459499932b2fcdd186d6 P 2b4519ad4d74462ebfa5eba155184701: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:40:04.435372 28522 log.cc:1196] T 83f68cf6db45459499932b2fcdd186d6 P 2b4519ad4d74462ebfa5eba155184701: Deleting WAL directory at /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/ts-1/wal/wals/83f68cf6db45459499932b2fcdd186d6
W20260929 18:40:04.435360 28013 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.100.1:42269 (error 108)
I20260929 18:40:04.435786 28522 ts_tablet_manager.cc:1939] T 83f68cf6db45459499932b2fcdd186d6 P 2b4519ad4d74462ebfa5eba155184701: Deleting consensus metadata
W20260929 18:40:04.436589 28497 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:40:04.436260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.1:42269 (local address 127.0.0.1:48288)
0929 18:40:04.436407 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0929 18:40:04.436480 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.1:42269: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":49}
I20260929 18:40:04.436766 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 28222
W20260929 18:40:04.438119 28013 catalog_manager.cc:4584] TS 9fe6f60bc87f4337a740ea25d9265b13 (127.13.100.1:42269): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 83f68cf6db45459499932b2fcdd186d6: Network error: Client connection negotiation failed: client connection to 127.13.100.1:42269: connect: Connection reset by peer (error 104)
I20260929 18:40:04.444195 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 28358
I20260929 18:40:04.446250 28497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:04.445929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.2:44645 (local address 127.0.0.1:36440)
0929 18:40:04.446072 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0929 18:40:04.446144 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.2:44645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260929 18:40:04.450179 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 27996
I20260929 18:40:04.456452 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.100.62:35827
--webserver_interface=127.13.100.62
--webserver_port=46467
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.62.keytab
--principal=kudu/127.13.100.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.100.20:43917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.100.62:35827
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:40:04.568761 28525 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:40:04.569060 28525 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:40:04.569104 28525 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:40:04.573307 28525 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:40:04.573372 28525 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:40:04.573395 28525 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:40:04.573415 28525 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:40:04.573434 28525 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:40:04.579104 28525 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.100.20:43917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.100.62:35827
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.62.keytab
--principal=kudu/127.13.100.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.100.62:35827
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.100.62
--webserver_port=46467
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28525
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:40:04.580091 28525 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:40:04.581044 28525 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:40:04.589329 28530 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:40:04.589329 28531 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:40:04.589370 28533 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:40:04.589613 28525 server_base.cc:1034] running on GCE node
I20260929 18:40:04.590165 28525 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:40:04.591245 28525 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:40:04.592428 28525 hybrid_clock.cc:648] HybridClock initialized: now 1790707204592423 us; error 35 us; skew 500 ppm
Sep 29 18:40:04 dist-test-slave-j1xs krb5kdc[27447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707204, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:40:04.595176 28525 init.cc:377] Logged in from keytab as kudu/127.13.100.62@KRBTEST.COM (short username kudu)
I20260929 18:40:04.596024 28525 webserver.cc:460] Webserver started at http://127.13.100.62:46467/ using document root <none> and password file <none>
I20260929 18:40:04.596566 28525 fs_manager.cc:359] Metadata directory not provided
I20260929 18:40:04.596652 28525 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:40:04.600361 28525 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:40:04.602512 28540 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:40:04.603619 28525 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:40:04.603747 28525 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/master-0/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/master-0/wal
uuid: "bc5af9cdcf5d4b16b729c8ac3ceb361c"
format_stamp: "Formatted at 2026-09-29 18:40:02 on dist-test-slave-j1xs"
server_key: "f78f34ab993a73e964650e60e2595883"
server_key_iv: "29027fd7bcfe9d91d74d334f1f49ab83"
server_key_version: "encryptionkey@0"
I20260929 18:40:04.604280 28525 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:40:04.619451 28525 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:40:04.622670 28525 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:40:04.622866 28525 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:40:04.630200 28525 rpc_server.cc:307] RPC server started. Bound to: 127.13.100.62:35827
I20260929 18:40:04.630488 28592 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.100.62:35827 every 8 connection(s)
I20260929 18:40:04.631404 28525 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
I20260929 18:40:04.632949 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 28525
I20260929 18:40:04.633435 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.100.1:42269
--local_ip_for_outbound_sockets=127.13.100.1
--tserver_master_addrs=127.13.100.62:35827
--webserver_port=38953
--webserver_interface=127.13.100.1
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.1.keytab
--principal=kudu/127.13.100.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.100.20:43917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 18:40:04.636217 28593 sys_catalog.cc:263] Verifying existing consensus state
I20260929 18:40:04.637854 28593 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc5af9cdcf5d4b16b729c8ac3ceb361c: Bootstrap starting.
I20260929 18:40:04.654985 28593 log.cc:824] T 00000000000000000000000000000000 P bc5af9cdcf5d4b16b729c8ac3ceb361c: Log is configured to *not* fsync() on all Append() calls
I20260929 18:40:04.665648 28593 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc5af9cdcf5d4b16b729c8ac3ceb361c: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 18:40:04.666059 28593 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc5af9cdcf5d4b16b729c8ac3ceb361c: Bootstrap complete.
I20260929 18:40:04.670017 28593 raft_consensus.cc:348] T 00000000000000000000000000000000 P bc5af9cdcf5d4b16b729c8ac3ceb361c [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc5af9cdcf5d4b16b729c8ac3ceb361c" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 35827 } }
I20260929 18:40:04.670392 28593 raft_consensus.cc:729] T 00000000000000000000000000000000 P bc5af9cdcf5d4b16b729c8ac3ceb361c [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc5af9cdcf5d4b16b729c8ac3ceb361c, State: Initialized, Role: FOLLOWER
I20260929 18:40:04.670861 28593 consensus_queue.cc:260] T 00000000000000000000000000000000 P bc5af9cdcf5d4b16b729c8ac3ceb361c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc5af9cdcf5d4b16b729c8ac3ceb361c" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 35827 } }
I20260929 18:40:04.670964 28593 raft_consensus.cc:388] T 00000000000000000000000000000000 P bc5af9cdcf5d4b16b729c8ac3ceb361c [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:40:04.671051 28593 raft_consensus.cc:482] T 00000000000000000000000000000000 P bc5af9cdcf5d4b16b729c8ac3ceb361c [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:40:04.671164 28593 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bc5af9cdcf5d4b16b729c8ac3ceb361c [term 2 FOLLOWER]: Advancing to term 3
I20260929 18:40:04.673050 28593 raft_consensus.cc:504] T 00000000000000000000000000000000 P bc5af9cdcf5d4b16b729c8ac3ceb361c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc5af9cdcf5d4b16b729c8ac3ceb361c" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 35827 } }
I20260929 18:40:04.673373 28593 leader_election.cc:302] T 00000000000000000000000000000000 P bc5af9cdcf5d4b16b729c8ac3ceb361c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bc5af9cdcf5d4b16b729c8ac3ceb361c; no voters:
I20260929 18:40:04.673693 28593 leader_election.cc:288] T 00000000000000000000000000000000 P bc5af9cdcf5d4b16b729c8ac3ceb361c [CANDIDATE]: Term 3 election: Requested vote from peers
I20260929 18:40:04.673806 28597 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bc5af9cdcf5d4b16b729c8ac3ceb361c [term 3 FOLLOWER]: Leader election won for term 3
I20260929 18:40:04.674077 28597 raft_consensus.cc:686] T 00000000000000000000000000000000 P bc5af9cdcf5d4b16b729c8ac3ceb361c [term 3 LEADER]: Becoming Leader. State: Replica: bc5af9cdcf5d4b16b729c8ac3ceb361c, State: Running, Role: LEADER
I20260929 18:40:04.674444 28597 consensus_queue.cc:237] T 00000000000000000000000000000000 P bc5af9cdcf5d4b16b729c8ac3ceb361c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc5af9cdcf5d4b16b729c8ac3ceb361c" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 35827 } }
I20260929 18:40:04.674863 28593 sys_catalog.cc:564] T 00000000000000000000000000000000 P bc5af9cdcf5d4b16b729c8ac3ceb361c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:40:04.675943 28598 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc5af9cdcf5d4b16b729c8ac3ceb361c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "bc5af9cdcf5d4b16b729c8ac3ceb361c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc5af9cdcf5d4b16b729c8ac3ceb361c" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 35827 } } }
I20260929 18:40:04.676065 28598 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc5af9cdcf5d4b16b729c8ac3ceb361c [sys.catalog]: This master's current role is: LEADER
I20260929 18:40:04.676704 28599 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc5af9cdcf5d4b16b729c8ac3ceb361c [sys.catalog]: SysCatalogTable state changed. Reason: New leader bc5af9cdcf5d4b16b729c8ac3ceb361c. Latest consensus state: current_term: 3 leader_uuid: "bc5af9cdcf5d4b16b729c8ac3ceb361c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc5af9cdcf5d4b16b729c8ac3ceb361c" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 35827 } } }
I20260929 18:40:04.676796 28599 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc5af9cdcf5d4b16b729c8ac3ceb361c [sys.catalog]: This master's current role is: LEADER
I20260929 18:40:04.679816 28613 catalog_manager.cc:1259] Loaded cluster ID: b85dd72d97cd477f92a9a3802549ebfa
I20260929 18:40:04.679879 28613 catalog_manager.cc:1552] T 00000000000000000000000000000000 P bc5af9cdcf5d4b16b729c8ac3ceb361c: loading cluster ID for follower catalog manager: success
W20260929 18:40:04.680820 28613 catalog_manager.cc:1576] T 00000000000000000000000000000000 P bc5af9cdcf5d4b16b729c8ac3ceb361c: acquiring CA information for follower catalog manager: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
W20260929 18:40:04.680895 28613 catalog_manager.cc:873] Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO: failed to prepare follower catalog manager, will retry
I20260929 18:40:04.680953 28606 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:40:04.683555 28606 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:40:04.684029 28606 catalog_manager.cc:1259] Loaded cluster ID: b85dd72d97cd477f92a9a3802549ebfa
I20260929 18:40:04.684085 28606 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260929 18:40:04.684654 28606 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7feef1a6a9ad google::LogMessage::Fail() at ??:0
@ 0x7feef1a6d88a google::LogMessage::SendToLog() at ??:0
@ 0x7feef1a6a4de google::LogMessage::Flush() at ??:0
@ 0x7feef1a6ba19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7feefc3a9fdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7feefc3aad2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7feefc3a7579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7feefc3eb640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7feef3cbc3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7feef3cbcc83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7feef3cbe3f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7feef3cad512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7feef52f16db start_thread at ??:0
@ 0x7feef056861f clone at ??:0
W20260929 18:40:04.783784 28595 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:40:04.784255 28595 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:40:04.784334 28595 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:40:04.790970 28595 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:40:04.791057 28595 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:40:04.791172 28595 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.100.1
I20260929 18:40:04.799721 28595 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.100.20:43917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.1.keytab
--principal=kudu/127.13.100.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.100.1:42269
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.100.1
--webserver_port=38953
--tserver_master_addrs=127.13.100.62:35827
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28595
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.100.1
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:40:04.801189 28595 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:40:04.802989 28595 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:40:04.814230 28634 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:40:04.814218 28635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:40:04.814201 28637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:40:04.814963 28595 server_base.cc:1034] running on GCE node
I20260929 18:40:04.815371 28595 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:40:04.816056 28595 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:40:04.817260 28595 hybrid_clock.cc:648] HybridClock initialized: now 1790707204817224 us; error 70 us; skew 500 ppm
Sep 29 18:40:04 dist-test-slave-j1xs krb5kdc[27447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707204, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:40:04.820230 28595 init.cc:377] Logged in from keytab as kudu/127.13.100.1@KRBTEST.COM (short username kudu)
I20260929 18:40:04.821141 28595 webserver.cc:460] Webserver started at http://127.13.100.1:38953/ using document root <none> and password file <none>
I20260929 18:40:04.821713 28595 fs_manager.cc:359] Metadata directory not provided
I20260929 18:40:04.821775 28595 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:40:04.825798 28595 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:40:04.828126 28644 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:40:04.829510 28595 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 18:40:04.829658 28595 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/ts-0/wal
uuid: "9fe6f60bc87f4337a740ea25d9265b13"
format_stamp: "Formatted at 2026-09-29 18:40:02 on dist-test-slave-j1xs"
server_key: "be8aa85caa3b213daed6b404d4b10721"
server_key_iv: "85d7fe3f3ffe0c297cd2dbbe62149376"
server_key_version: "encryptionkey@0"
I20260929 18:40:04.830376 28595 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:40:04.858271 28595 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:40:04.861785 28595 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:40:04.862048 28595 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:40:04.862671 28595 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:40:04.863837 28651 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260929 18:40:04.871131 28595 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 18:40:04.871243 28595 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.008s user 0.000s sys 0.000s
I20260929 18:40:04.871333 28595 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 18:40:04.873037 28595 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 18:40:04.873122 28595 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260929 18:40:04.873445 28651 tablet_bootstrap.cc:492] T 83f68cf6db45459499932b2fcdd186d6 P 9fe6f60bc87f4337a740ea25d9265b13: Bootstrap starting.
I20260929 18:40:04.883764 28595 rpc_server.cc:307] RPC server started. Bound to: 127.13.100.1:42269
I20260929 18:40:04.884042 28758 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.100.1:42269 every 8 connection(s)
I20260929 18:40:04.884838 28595 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/ts-0/data/info.pb
I20260929 18:40:04.891029 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 28595
I20260929 18:40:04.891515 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.100.2:44645
--local_ip_for_outbound_sockets=127.13.100.2
--tserver_master_addrs=127.13.100.62:35827
--webserver_port=46185
--webserver_interface=127.13.100.2
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.2.keytab
--principal=kudu/127.13.100.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.100.20:43917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 18:40:04.897428 28651 log.cc:824] T 83f68cf6db45459499932b2fcdd186d6 P 9fe6f60bc87f4337a740ea25d9265b13: Log is configured to *not* fsync() on all Append() calls
I20260929 18:40:04.905050 28651 tablet_bootstrap.cc:492] T 83f68cf6db45459499932b2fcdd186d6 P 9fe6f60bc87f4337a740ea25d9265b13: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 18:40:04.905498 28651 tablet_bootstrap.cc:492] T 83f68cf6db45459499932b2fcdd186d6 P 9fe6f60bc87f4337a740ea25d9265b13: Bootstrap complete.
I20260929 18:40:04.906703 28651 ts_tablet_manager.cc:1397] T 83f68cf6db45459499932b2fcdd186d6 P 9fe6f60bc87f4337a740ea25d9265b13: Time spent bootstrapping tablet: real 0.034s user 0.017s sys 0.013s
I20260929 18:40:04.909301 28651 raft_consensus.cc:348] T 83f68cf6db45459499932b2fcdd186d6 P 9fe6f60bc87f4337a740ea25d9265b13 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b4519ad4d74462ebfa5eba155184701" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 44645 } } peers { permanent_uuid: "a5aa041fdd8a4573b36ee7aee2085658" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 45395 } } peers { permanent_uuid: "9fe6f60bc87f4337a740ea25d9265b13" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 42269 } }
I20260929 18:40:04.909747 28651 raft_consensus.cc:729] T 83f68cf6db45459499932b2fcdd186d6 P 9fe6f60bc87f4337a740ea25d9265b13 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fe6f60bc87f4337a740ea25d9265b13, State: Initialized, Role: FOLLOWER
I20260929 18:40:04.910279 28651 consensus_queue.cc:260] T 83f68cf6db45459499932b2fcdd186d6 P 9fe6f60bc87f4337a740ea25d9265b13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b4519ad4d74462ebfa5eba155184701" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 44645 } } peers { permanent_uuid: "a5aa041fdd8a4573b36ee7aee2085658" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 45395 } } peers { permanent_uuid: "9fe6f60bc87f4337a740ea25d9265b13" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 42269 } }
I20260929 18:40:04.911255 28651 ts_tablet_manager.cc:1428] T 83f68cf6db45459499932b2fcdd186d6 P 9fe6f60bc87f4337a740ea25d9265b13: Time spent starting tablet: real 0.004s user 0.004s sys 0.001s
W20260929 18:40:04.973297 28761 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:40:04.886876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:35827 (local address 127.13.100.1:38967)
0929 18:40:04.887344 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0929 18:40:04.887395 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 18:40:04.888218 (+ 823us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:40:04.972533 (+ 84315us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.62:35827: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":135,"threads_started":1}
W20260929 18:40:04.973769 28759 heartbeater.cc:644] Failed to heartbeat to 127.13.100.62:35827 (0 consecutive failures): Network error: Failed to ping master at 127.13.100.62:35827: Client connection negotiation failed: client connection to 127.13.100.62:35827: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 18:40:04.974539 28761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:04.974268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:35827 (local address 127.13.100.1:47149)
0929 18:40:04.974394 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0929 18:40:04.974476 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.62:35827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260929 18:40:04.975157 28761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:04.974949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:35827 (local address 127.13.100.1:34535)
0929 18:40:04.975052 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0929 18:40:04.975104 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.62:35827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260929 18:40:04.975315 28759 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 18:40:05.033124 28762 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:40:05.033438 28762 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:40:05.033524 28762 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:40:05.037513 28762 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:40:05.037600 28762 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:40:05.037693 28762 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.100.2
I20260929 18:40:05.042837 28762 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.100.20:43917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.2.keytab
--principal=kudu/127.13.100.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.100.2:44645
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.100.2
--webserver_port=46185
--tserver_master_addrs=127.13.100.62:35827
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28762
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.100.2
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:40:05.043893 28762 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:40:05.044914 28762 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:40:05.054242 28772 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:40:05.054248 28770 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:40:05.054257 28769 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:40:05.054494 28762 server_base.cc:1034] running on GCE node
I20260929 18:40:05.054877 28762 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:40:05.055495 28762 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:40:05.056670 28762 hybrid_clock.cc:648] HybridClock initialized: now 1790707205056635 us; error 45 us; skew 500 ppm
Sep 29 18:40:05 dist-test-slave-j1xs krb5kdc[27447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707205, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:40:05.059465 28762 init.cc:377] Logged in from keytab as kudu/127.13.100.2@KRBTEST.COM (short username kudu)
I20260929 18:40:05.060328 28762 webserver.cc:460] Webserver started at http://127.13.100.2:46185/ using document root <none> and password file <none>
I20260929 18:40:05.060865 28762 fs_manager.cc:359] Metadata directory not provided
I20260929 18:40:05.060927 28762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:40:05.064750 28762 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.002s
I20260929 18:40:05.066844 28779 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:40:05.068094 28762 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:40:05.068222 28762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/ts-1/wal
uuid: "2b4519ad4d74462ebfa5eba155184701"
format_stamp: "Formatted at 2026-09-29 18:40:02 on dist-test-slave-j1xs"
server_key: "bce4f1c80ce470c76043a3749f9a2130"
server_key_iv: "c3668ad0ef7af0f4ace653617c9fb506"
server_key_version: "encryptionkey@0"
I20260929 18:40:05.068758 28762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:40:05.091681 28762 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:40:05.094624 28762 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:40:05.094841 28762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:40:05.095441 28762 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:40:05.096243 28762 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:40:05.096311 28762 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:40:05.096381 28762 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:40:05.096429 28762 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:40:05.105588 28762 rpc_server.cc:307] RPC server started. Bound to: 127.13.100.2:44645
I20260929 18:40:05.105669 28892 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.100.2:44645 every 8 connection(s)
I20260929 18:40:05.106616 28762 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/ts-1/data/info.pb
I20260929 18:40:05.109117 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 28762
I20260929 18:40:05.109573 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.100.3:45395
--local_ip_for_outbound_sockets=127.13.100.3
--tserver_master_addrs=127.13.100.62:35827
--webserver_port=36755
--webserver_interface=127.13.100.3
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.3.keytab
--principal=kudu/127.13.100.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.100.20:43917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 18:40:05.110291 28895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:05.108714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:35827 (local address 127.13.100.2:36487)
0929 18:40:05.109180 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0929 18:40:05.109305 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.62:35827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":138,"threads_started":1}
W20260929 18:40:05.110720 28893 heartbeater.cc:644] Failed to heartbeat to 127.13.100.62:35827 (0 consecutive failures): Network error: Failed to ping master at 127.13.100.62:35827: Client connection negotiation failed: client connection to 127.13.100.62:35827: connect: Connection refused (error 111)
I20260929 18:40:05.111268 28895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:05.111065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:35827 (local address 127.13.100.2:44119)
0929 18:40:05.111168 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0929 18:40:05.111218 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.62:35827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260929 18:40:05.111966 28895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:05.111772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:35827 (local address 127.13.100.2:46433)
0929 18:40:05.111873 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0929 18:40:05.111912 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.62:35827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260929 18:40:05.112187 28893 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 18:40:05.226615 28896 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:40:05.226872 28896 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:40:05.226917 28896 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:40:05.230849 28896 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:40:05.230909 28896 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:40:05.230967 28896 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.100.3
I20260929 18:40:05.236387 28896 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.100.20:43917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.3.keytab
--principal=kudu/127.13.100.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.100.3:45395
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.100.3
--webserver_port=36755
--tserver_master_addrs=127.13.100.62:35827
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28896
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.100.3
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:40:05.237401 28896 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:40:05.238401 28896 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 18:40:05.249125 28896 server_base.cc:1034] running on GCE node
W20260929 18:40:05.249058 28901 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:40:05.249058 28904 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:40:05.249076 28902 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:40:05.249825 28896 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:40:05.250629 28896 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:40:05.251839 28896 hybrid_clock.cc:648] HybridClock initialized: now 1790707205251803 us; error 47 us; skew 500 ppm
Sep 29 18:40:05 dist-test-slave-j1xs krb5kdc[27447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707205, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:40:05.255077 28896 init.cc:377] Logged in from keytab as kudu/127.13.100.3@KRBTEST.COM (short username kudu)
I20260929 18:40:05.256035 28896 webserver.cc:460] Webserver started at http://127.13.100.3:36755/ using document root <none> and password file <none>
I20260929 18:40:05.256618 28896 fs_manager.cc:359] Metadata directory not provided
I20260929 18:40:05.256705 28896 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:40:05.261143 28896 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 18:40:05.263726 28911 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:40:05.265364 28896 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 18:40:05.265533 28896 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/ts-2/wal
uuid: "a5aa041fdd8a4573b36ee7aee2085658"
format_stamp: "Formatted at 2026-09-29 18:40:02 on dist-test-slave-j1xs"
server_key: "351f8f834e1c4853a9ef0f872f19da00"
server_key_iv: "8a1dbdae6e6a05b37f329313c66820da"
server_key_version: "encryptionkey@0"
I20260929 18:40:05.266199 28896 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:40:05.288033 28896 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:40:05.291640 28896 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:40:05.291884 28896 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:40:05.292541 28896 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:40:05.293402 28896 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:40:05.293469 28896 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:40:05.293534 28896 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:40:05.293581 28896 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:40:05.303886 28896 rpc_server.cc:307] RPC server started. Bound to: 127.13.100.3:45395
I20260929 18:40:05.303908 29024 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.100.3:45395 every 8 connection(s)
I20260929 18:40:05.304929 28896 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707036041078-13712-0/minicluster-data/ts-2/data/info.pb
I20260929 18:40:05.307082 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 28896
I20260929 18:40:05.309026 29027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:05.307273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:35827 (local address 127.13.100.3:43413)
0929 18:40:05.307811 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0929 18:40:05.307944 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.62:35827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":158,"threads_started":1}
W20260929 18:40:05.309441 29025 heartbeater.cc:644] Failed to heartbeat to 127.13.100.62:35827 (0 consecutive failures): Network error: Failed to ping master at 127.13.100.62:35827: Client connection negotiation failed: client connection to 127.13.100.62:35827: connect: Connection refused (error 111)
I20260929 18:40:05.310056 29027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:05.309812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:35827 (local address 127.13.100.3:45609)
0929 18:40:05.309936 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0929 18:40:05.309997 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.62:35827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260929 18:40:05.310623 29027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:05.310411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:35827 (local address 127.13.100.3:37045)
0929 18:40:05.310514 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0929 18:40:05.310562 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.62:35827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260929 18:40:05.310789 29025 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 18:40:05.976382 29029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:05.975944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:35827 (local address 127.13.100.1:60637)
0929 18:40:05.976211 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0929 18:40:05.976269 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.62:35827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":72,"threads_started":1}
I20260929 18:40:06.113255 29030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:06.112814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:35827 (local address 127.13.100.2:53203)
0929 18:40:06.113083 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0929 18:40:06.113141 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.62:35827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":69,"threads_started":1}
I20260929 18:40:06.311705 29031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:06.311330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:35827 (local address 127.13.100.3:42567)
0929 18:40:06.311551 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0929 18:40:06.311607 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.62:35827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":68,"threads_started":1}
I20260929 18:40:06.569741 29032 raft_consensus.cc:482] T 83f68cf6db45459499932b2fcdd186d6 P 9fe6f60bc87f4337a740ea25d9265b13 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:40:06.570051 29032 raft_consensus.cc:504] T 83f68cf6db45459499932b2fcdd186d6 P 9fe6f60bc87f4337a740ea25d9265b13 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b4519ad4d74462ebfa5eba155184701" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 44645 } } peers { permanent_uuid: "a5aa041fdd8a4573b36ee7aee2085658" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 45395 } } peers { permanent_uuid: "9fe6f60bc87f4337a740ea25d9265b13" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 42269 } }
I20260929 18:40:06.571269 29032 leader_election.cc:288] T 83f68cf6db45459499932b2fcdd186d6 P 9fe6f60bc87f4337a740ea25d9265b13 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2b4519ad4d74462ebfa5eba155184701 (127.13.100.2:44645), a5aa041fdd8a4573b36ee7aee2085658 (127.13.100.3:45395)
Sep 29 18:40:06 dist-test-slave-j1xs krb5kdc[27447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707204, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.1@KRBTEST.COM for kudu/127.13.100.3@KRBTEST.COM
Sep 29 18:40:06 dist-test-slave-j1xs krb5kdc[27447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707204, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.1@KRBTEST.COM for kudu/127.13.100.2@KRBTEST.COM
I20260929 18:40:06.588430 29033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:06.571597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.3:45395 (local address 127.13.100.1:58419)
0929 18:40:06.571845 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0929 18:40:06.571864 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 18:40:06.571998 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:40:06.573538 (+ 1540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:40:06.573545 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:40:06.574133 (+ 588us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:40:06.574624 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:40:06.574637 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:40:06.575960 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:40:06.575964 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:40:06.576892 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:40:06.576902 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:40:06.577817 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:40:06.577827 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:40:06.578017 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:40:06.581751 (+ 3734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:40:06.581835 (+ 84us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:40:06.583658 (+ 1823us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:40:06.586003 (+ 2345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:40:06.586010 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:40:06.586023 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:40:06.586278 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:40:06.586625 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:40:06.586629 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:40:06.586631 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:40:06.586763 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:40:06.587455 (+ 692us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:40:06.587462 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:40:06.587683 (+ 221us) client_negotiation.cc:769] Sending connection context
0929 18:40:06.587863 (+ 180us) client_negotiation.cc:240] Negotiation successful
0929 18:40:06.588221 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"mutex_wait_us":39,"spinlock_wait_cycles":6528,"thread_start_us":80,"threads_started":1}
I20260929 18:40:06.588435 29035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:06.571777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.1:43907 (local address 127.13.100.2:44645)
0929 18:40:06.572223 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0929 18:40:06.572250 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0929 18:40:06.572319 (+ 69us) server_negotiation.cc:407] Connection header received
0929 18:40:06.573181 (+ 862us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:40:06.573189 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:40:06.573538 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:40:06.573826 (+ 288us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:40:06.574830 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:06.575802 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:06.577016 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:06.577672 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:06.577857 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:40:06.584255 (+ 6398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:40:06.584260 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:40:06.584271 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:40:06.584299 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:40:06.586166 (+ 1867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:40:06.586614 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:40:06.586624 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:40:06.586628 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:40:06.586728 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:40:06.587025 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:40:06.587028 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:40:06.587030 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:40:06.587384 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:40:06.587525 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0929 18:40:06.587965 (+ 440us) server_negotiation.cc:299] Negotiation successful
0929 18:40:06.588240 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":106,"threads_started":1}
I20260929 18:40:06.588469 29036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:06.571597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.2:44645 (local address 127.13.100.1:43907)
0929 18:40:06.572226 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0929 18:40:06.572242 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 18:40:06.572345 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:40:06.573803 (+ 1458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:40:06.573806 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:40:06.574155 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:40:06.574621 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:40:06.574635 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:40:06.575963 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:40:06.575966 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:40:06.576896 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:40:06.576905 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:40:06.578075 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:40:06.578078 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:40:06.578146 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:40:06.581749 (+ 3603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:40:06.581777 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:40:06.584058 (+ 2281us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:40:06.586282 (+ 2224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:40:06.586289 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:40:06.586291 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:40:06.586501 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:40:06.586829 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:40:06.586832 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:40:06.586834 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:40:06.586915 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:40:06.587489 (+ 574us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:40:06.587492 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:40:06.587683 (+ 191us) client_negotiation.cc:769] Sending connection context
0929 18:40:06.587863 (+ 180us) client_negotiation.cc:240] Negotiation successful
0929 18:40:06.588299 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"mutex_wait_us":721,"spinlock_wait_cycles":2944,"thread_start_us":61,"threads_started":1}
I20260929 18:40:06.589283 29034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:06.571755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.1:58419 (local address 127.13.100.3:45395)
0929 18:40:06.572017 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0929 18:40:06.572042 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0929 18:40:06.572091 (+ 49us) server_negotiation.cc:407] Connection header received
0929 18:40:06.572786 (+ 695us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:40:06.572792 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:40:06.573195 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:40:06.573434 (+ 239us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:40:06.574822 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:06.575806 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:06.577066 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:06.577669 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:06.578385 (+ 716us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:40:06.583915 (+ 5530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:40:06.583920 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:40:06.583933 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:40:06.583961 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:40:06.585854 (+ 1893us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:40:06.586395 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:40:06.586402 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:40:06.586407 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:40:06.586523 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:40:06.586930 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:40:06.586936 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:40:06.586937 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:40:06.587344 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:40:06.587480 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0929 18:40:06.588904 (+ 1424us) server_negotiation.cc:299] Negotiation successful
0929 18:40:06.589126 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":84,"threads_started":1}
W20260929 18:40:06.591560 28646 leader_election.cc:341] T 83f68cf6db45459499932b2fcdd186d6 P 9fe6f60bc87f4337a740ea25d9265b13 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a5aa041fdd8a4573b36ee7aee2085658 (127.13.100.3:45395): Not found: Tablet not found: 83f68cf6db45459499932b2fcdd186d6
W20260929 18:40:06.591718 28648 leader_election.cc:341] T 83f68cf6db45459499932b2fcdd186d6 P 9fe6f60bc87f4337a740ea25d9265b13 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2b4519ad4d74462ebfa5eba155184701 (127.13.100.2:44645): Not found: Tablet not found: 83f68cf6db45459499932b2fcdd186d6
I20260929 18:40:06.591795 28648 leader_election.cc:302] T 83f68cf6db45459499932b2fcdd186d6 P 9fe6f60bc87f4337a740ea25d9265b13 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9fe6f60bc87f4337a740ea25d9265b13; no voters: 2b4519ad4d74462ebfa5eba155184701, a5aa041fdd8a4573b36ee7aee2085658
I20260929 18:40:06.592010 29032 raft_consensus.cc:2726] T 83f68cf6db45459499932b2fcdd186d6 P 9fe6f60bc87f4337a740ea25d9265b13 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 18:40:06.977365 29036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:06.977113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:35827 (local address 127.13.100.1:48785)
0929 18:40:06.977232 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0929 18:40:06.977285 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.62:35827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260929 18:40:07.114380 29037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:07.114005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:35827 (local address 127.13.100.2:44955)
0929 18:40:07.114240 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0929 18:40:07.114295 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.62:35827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":65,"threads_started":1}
I20260929 18:40:07.312752 29038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:07.312397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:35827 (local address 127.13.100.3:58739)
0929 18:40:07.312618 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0929 18:40:07.312670 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.62:35827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":70,"threads_started":1}
I20260929 18:40:07.978717 29039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:07.978259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:35827 (local address 127.13.100.1:38935)
0929 18:40:07.978514 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0929 18:40:07.978591 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.62:35827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":76,"threads_started":1}
I20260929 18:40:08.115515 29040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:08.115110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:35827 (local address 127.13.100.2:48015)
0929 18:40:08.115351 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0929 18:40:08.115408 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.62:35827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":76,"threads_started":1}
I20260929 18:40:08.313982 29041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:08.313492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:35827 (local address 127.13.100.3:39241)
0929 18:40:08.313750 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0929 18:40:08.313809 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.62:35827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":90,"threads_started":1}
I20260929 18:40:08.416488 29042 raft_consensus.cc:482] T 83f68cf6db45459499932b2fcdd186d6 P 9fe6f60bc87f4337a740ea25d9265b13 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:40:08.416654 29042 raft_consensus.cc:504] T 83f68cf6db45459499932b2fcdd186d6 P 9fe6f60bc87f4337a740ea25d9265b13 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b4519ad4d74462ebfa5eba155184701" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 44645 } } peers { permanent_uuid: "a5aa041fdd8a4573b36ee7aee2085658" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 45395 } } peers { permanent_uuid: "9fe6f60bc87f4337a740ea25d9265b13" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 42269 } }
I20260929 18:40:08.417145 29042 leader_election.cc:288] T 83f68cf6db45459499932b2fcdd186d6 P 9fe6f60bc87f4337a740ea25d9265b13 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2b4519ad4d74462ebfa5eba155184701 (127.13.100.2:44645), a5aa041fdd8a4573b36ee7aee2085658 (127.13.100.3:45395)
W20260929 18:40:08.417918 28648 leader_election.cc:341] T 83f68cf6db45459499932b2fcdd186d6 P 9fe6f60bc87f4337a740ea25d9265b13 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2b4519ad4d74462ebfa5eba155184701 (127.13.100.2:44645): Not found: Tablet not found: 83f68cf6db45459499932b2fcdd186d6
W20260929 18:40:08.418046 28646 leader_election.cc:341] T 83f68cf6db45459499932b2fcdd186d6 P 9fe6f60bc87f4337a740ea25d9265b13 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a5aa041fdd8a4573b36ee7aee2085658 (127.13.100.3:45395): Not found: Tablet not found: 83f68cf6db45459499932b2fcdd186d6
I20260929 18:40:08.418145 28646 leader_election.cc:302] T 83f68cf6db45459499932b2fcdd186d6 P 9fe6f60bc87f4337a740ea25d9265b13 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9fe6f60bc87f4337a740ea25d9265b13; no voters: 2b4519ad4d74462ebfa5eba155184701, a5aa041fdd8a4573b36ee7aee2085658
I20260929 18:40:08.418337 29042 raft_consensus.cc:2726] T 83f68cf6db45459499932b2fcdd186d6 P 9fe6f60bc87f4337a740ea25d9265b13 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 18:40:08.979898 29043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:08.979477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:35827 (local address 127.13.100.1:56861)
0929 18:40:08.979743 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0929 18:40:08.979799 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.62:35827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":73,"threads_started":1}
I20260929 18:40:09.116598 29044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:09.116209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:35827 (local address 127.13.100.2:58829)
0929 18:40:09.116454 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0929 18:40:09.116509 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.62:35827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":70,"threads_started":1}
I20260929 18:40:09.315336 29045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:09.314789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:35827 (local address 127.13.100.3:44639)
0929 18:40:09.315156 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0929 18:40:09.315218 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.62:35827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":76,"threads_started":1}
I20260929 18:40:09.981329 29046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:09.980876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:35827 (local address 127.13.100.1:37229)
0929 18:40:09.981173 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0929 18:40:09.981233 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.62:35827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":79,"threads_started":1}
I20260929 18:40:10.117853 29047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:10.117411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:35827 (local address 127.13.100.2:50119)
0929 18:40:10.117698 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0929 18:40:10.117760 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.62:35827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":79,"threads_started":1}
I20260929 18:40:10.316638 29048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:10.316232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:35827 (local address 127.13.100.3:39283)
0929 18:40:10.316497 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0929 18:40:10.316550 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.62:35827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":92,"threads_started":1}
I20260929 18:40:10.646729 29049 raft_consensus.cc:482] T 83f68cf6db45459499932b2fcdd186d6 P 9fe6f60bc87f4337a740ea25d9265b13 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:40:10.646860 29049 raft_consensus.cc:504] T 83f68cf6db45459499932b2fcdd186d6 P 9fe6f60bc87f4337a740ea25d9265b13 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b4519ad4d74462ebfa5eba155184701" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 44645 } } peers { permanent_uuid: "a5aa041fdd8a4573b36ee7aee2085658" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 45395 } } peers { permanent_uuid: "9fe6f60bc87f4337a740ea25d9265b13" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 42269 } }
I20260929 18:40:10.647338 29049 leader_election.cc:288] T 83f68cf6db45459499932b2fcdd186d6 P 9fe6f60bc87f4337a740ea25d9265b13 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2b4519ad4d74462ebfa5eba155184701 (127.13.100.2:44645), a5aa041fdd8a4573b36ee7aee2085658 (127.13.100.3:45395)
W20260929 18:40:10.648281 28648 leader_election.cc:341] T 83f68cf6db45459499932b2fcdd186d6 P 9fe6f60bc87f4337a740ea25d9265b13 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2b4519ad4d74462ebfa5eba155184701 (127.13.100.2:44645): Not found: Tablet not found: 83f68cf6db45459499932b2fcdd186d6
W20260929 18:40:10.648367 28646 leader_election.cc:341] T 83f68cf6db45459499932b2fcdd186d6 P 9fe6f60bc87f4337a740ea25d9265b13 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a5aa041fdd8a4573b36ee7aee2085658 (127.13.100.3:45395): Not found: Tablet not found: 83f68cf6db45459499932b2fcdd186d6
I20260929 18:40:10.648427 28646 leader_election.cc:302] T 83f68cf6db45459499932b2fcdd186d6 P 9fe6f60bc87f4337a740ea25d9265b13 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9fe6f60bc87f4337a740ea25d9265b13; no voters: 2b4519ad4d74462ebfa5eba155184701, a5aa041fdd8a4573b36ee7aee2085658
I20260929 18:40:10.648604 29049 raft_consensus.cc:2726] T 83f68cf6db45459499932b2fcdd186d6 P 9fe6f60bc87f4337a740ea25d9265b13 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 18:40:10.982626 29050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:10.982204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:35827 (local address 127.13.100.1:44435)
0929 18:40:10.982454 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0929 18:40:10.982508 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.62:35827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":91,"threads_started":1}
I20260929 18:40:11.119017 29051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:11.118603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:35827 (local address 127.13.100.2:42295)
0929 18:40:11.118845 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0929 18:40:11.118927 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.62:35827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":82,"threads_started":1}
I20260929 18:40:11.317934 29052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:11.317447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:35827 (local address 127.13.100.3:59673)
0929 18:40:11.317723 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0929 18:40:11.317793 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.62:35827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":73,"threads_started":1}
I20260929 18:40:11.983754 29053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:11.983341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:35827 (local address 127.13.100.1:47635)
0929 18:40:11.983603 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0929 18:40:11.983662 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.62:35827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":83,"threads_started":1}
I20260929 18:40:12.120214 29054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:12.119802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:35827 (local address 127.13.100.2:37337)
0929 18:40:12.120071 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0929 18:40:12.120125 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.62:35827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":72,"threads_started":1}
I20260929 18:40:12.319187 29055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:12.318714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:35827 (local address 127.13.100.3:44807)
0929 18:40:12.319036 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0929 18:40:12.319095 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.62:35827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":77,"threads_started":1}
I20260929 18:40:12.984978 29056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:12.984529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:35827 (local address 127.13.100.1:47985)
0929 18:40:12.984820 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0929 18:40:12.984883 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.62:35827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":94,"threads_started":1}
I20260929 18:40:13.121438 29057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:13.120988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:35827 (local address 127.13.100.2:34523)
0929 18:40:13.121261 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0929 18:40:13.121321 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.62:35827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":84,"threads_started":1}
I20260929 18:40:13.320483 29058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:13.320045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:35827 (local address 127.13.100.3:57121)
0929 18:40:13.320317 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0929 18:40:13.320385 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.62:35827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":111,"threads_started":1}
I20260929 18:40:13.534929 29059 raft_consensus.cc:482] T 83f68cf6db45459499932b2fcdd186d6 P 9fe6f60bc87f4337a740ea25d9265b13 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:40:13.535082 29059 raft_consensus.cc:504] T 83f68cf6db45459499932b2fcdd186d6 P 9fe6f60bc87f4337a740ea25d9265b13 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b4519ad4d74462ebfa5eba155184701" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 44645 } } peers { permanent_uuid: "a5aa041fdd8a4573b36ee7aee2085658" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 45395 } } peers { permanent_uuid: "9fe6f60bc87f4337a740ea25d9265b13" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 42269 } }
I20260929 18:40:13.535490 29059 leader_election.cc:288] T 83f68cf6db45459499932b2fcdd186d6 P 9fe6f60bc87f4337a740ea25d9265b13 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2b4519ad4d74462ebfa5eba155184701 (127.13.100.2:44645), a5aa041fdd8a4573b36ee7aee2085658 (127.13.100.3:45395)
W20260929 18:40:13.536243 28646 leader_election.cc:341] T 83f68cf6db45459499932b2fcdd186d6 P 9fe6f60bc87f4337a740ea25d9265b13 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a5aa041fdd8a4573b36ee7aee2085658 (127.13.100.3:45395): Not found: Tablet not found: 83f68cf6db45459499932b2fcdd186d6
W20260929 18:40:13.536355 28648 leader_election.cc:341] T 83f68cf6db45459499932b2fcdd186d6 P 9fe6f60bc87f4337a740ea25d9265b13 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2b4519ad4d74462ebfa5eba155184701 (127.13.100.2:44645): Not found: Tablet not found: 83f68cf6db45459499932b2fcdd186d6
I20260929 18:40:13.536474 28648 leader_election.cc:302] T 83f68cf6db45459499932b2fcdd186d6 P 9fe6f60bc87f4337a740ea25d9265b13 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9fe6f60bc87f4337a740ea25d9265b13; no voters: 2b4519ad4d74462ebfa5eba155184701, a5aa041fdd8a4573b36ee7aee2085658
I20260929 18:40:13.536661 29059 raft_consensus.cc:2726] T 83f68cf6db45459499932b2fcdd186d6 P 9fe6f60bc87f4337a740ea25d9265b13 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 18:40:13.986325 29060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:13.985811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:35827 (local address 127.13.100.1:56583)
0929 18:40:13.986156 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0929 18:40:13.986221 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.62:35827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":169,"threads_started":1}
I20260929 18:40:14.122648 29061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:14.122237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:35827 (local address 127.13.100.2:45735)
0929 18:40:14.122493 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0929 18:40:14.122553 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.62:35827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":76,"threads_started":1}
I20260929 18:40:14.321760 29062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:14.321275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:35827 (local address 127.13.100.3:48401)
0929 18:40:14.321608 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0929 18:40:14.321667 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.62:35827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":99,"threads_started":1}
I20260929 18:40:14.987550 29063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:14.987152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:35827 (local address 127.13.100.1:55303)
0929 18:40:14.987395 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0929 18:40:14.987449 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.62:35827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":80,"threads_started":1}
I20260929 18:40:15.123840 29064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:15.123453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:35827 (local address 127.13.100.2:60937)
0929 18:40:15.123695 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0929 18:40:15.123751 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.62:35827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":84,"threads_started":1}
I20260929 18:40:15.322999 29065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:15.322598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:35827 (local address 127.13.100.3:55103)
0929 18:40:15.322849 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0929 18:40:15.322906 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.62:35827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":80,"threads_started":1}
I20260929 18:40:15.988776 29066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:15.988372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:35827 (local address 127.13.100.1:57293)
0929 18:40:15.988620 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0929 18:40:15.988675 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.62:35827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":72,"threads_started":1}
I20260929 18:40:16.125165 29067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:16.124642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:35827 (local address 127.13.100.2:57401)
0929 18:40:16.124924 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0929 18:40:16.125064 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.62:35827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":74,"threads_started":1}
I20260929 18:40:16.324120 29068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:16.323733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:35827 (local address 127.13.100.3:49945)
0929 18:40:16.323972 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0929 18:40:16.324029 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.62:35827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":70,"threads_started":1}
I20260929 18:40:16.990034 29069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:16.989491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:35827 (local address 127.13.100.1:48461)
0929 18:40:16.989778 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0929 18:40:16.989831 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.62:35827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":83,"threads_started":1}
I20260929 18:40:17.126693 29070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:17.126236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:35827 (local address 127.13.100.2:43887)
0929 18:40:17.126512 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0929 18:40:17.126574 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.62:35827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":75,"threads_started":1}
I20260929 18:40:17.325222 29071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:17.324826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:35827 (local address 127.13.100.3:44199)
0929 18:40:17.325072 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0929 18:40:17.325130 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.62:35827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":69,"threads_started":1}
I20260929 18:40:17.991379 29072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:17.990866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:35827 (local address 127.13.100.1:40485)
0929 18:40:17.991159 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0929 18:40:17.991229 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.62:35827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":72,"threads_started":1}
I20260929 18:40:18.127961 29073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:18.127552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:35827 (local address 127.13.100.2:34785)
0929 18:40:18.127804 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0929 18:40:18.127865 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.62:35827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":73,"threads_started":1}
I20260929 18:40:18.326610 29074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:18.326174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:35827 (local address 127.13.100.3:41175)
0929 18:40:18.326440 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0929 18:40:18.326505 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.62:35827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":78,"threads_started":1}
I20260929 18:40:18.536885 29075 raft_consensus.cc:482] T 83f68cf6db45459499932b2fcdd186d6 P 9fe6f60bc87f4337a740ea25d9265b13 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:40:18.537010 29075 raft_consensus.cc:504] T 83f68cf6db45459499932b2fcdd186d6 P 9fe6f60bc87f4337a740ea25d9265b13 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b4519ad4d74462ebfa5eba155184701" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 44645 } } peers { permanent_uuid: "a5aa041fdd8a4573b36ee7aee2085658" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 45395 } } peers { permanent_uuid: "9fe6f60bc87f4337a740ea25d9265b13" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 42269 } }
I20260929 18:40:18.537431 29075 leader_election.cc:288] T 83f68cf6db45459499932b2fcdd186d6 P 9fe6f60bc87f4337a740ea25d9265b13 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2b4519ad4d74462ebfa5eba155184701 (127.13.100.2:44645), a5aa041fdd8a4573b36ee7aee2085658 (127.13.100.3:45395)
W20260929 18:40:18.538214 28648 leader_election.cc:341] T 83f68cf6db45459499932b2fcdd186d6 P 9fe6f60bc87f4337a740ea25d9265b13 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2b4519ad4d74462ebfa5eba155184701 (127.13.100.2:44645): Not found: Tablet not found: 83f68cf6db45459499932b2fcdd186d6
W20260929 18:40:18.538323 28646 leader_election.cc:341] T 83f68cf6db45459499932b2fcdd186d6 P 9fe6f60bc87f4337a740ea25d9265b13 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a5aa041fdd8a4573b36ee7aee2085658 (127.13.100.3:45395): Not found: Tablet not found: 83f68cf6db45459499932b2fcdd186d6
I20260929 18:40:18.538440 28646 leader_election.cc:302] T 83f68cf6db45459499932b2fcdd186d6 P 9fe6f60bc87f4337a740ea25d9265b13 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9fe6f60bc87f4337a740ea25d9265b13; no voters: 2b4519ad4d74462ebfa5eba155184701, a5aa041fdd8a4573b36ee7aee2085658
I20260929 18:40:18.538604 29075 raft_consensus.cc:2726] T 83f68cf6db45459499932b2fcdd186d6 P 9fe6f60bc87f4337a740ea25d9265b13 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 18:40:18.992726 29076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:18.992262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:35827 (local address 127.13.100.1:42479)
0929 18:40:18.992554 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0929 18:40:18.992618 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.62:35827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":75,"threads_started":1}
I20260929 18:40:19.129730 29077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:19.129160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:35827 (local address 127.13.100.2:38399)
0929 18:40:19.129512 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0929 18:40:19.129589 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.62:35827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":117,"threads_started":1}
I20260929 18:40:19.327749 29078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:19.327372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:35827 (local address 127.13.100.3:41749)
0929 18:40:19.327602 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0929 18:40:19.327655 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.62:35827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":74,"threads_started":1}
I20260929 18:40:19.994153 29079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:19.993686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:35827 (local address 127.13.100.1:47789)
0929 18:40:19.993984 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0929 18:40:19.994042 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.62:35827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":71,"threads_started":1}
I20260929 18:40:20.131062 29080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:20.130660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:35827 (local address 127.13.100.2:45117)
0929 18:40:20.130912 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0929 18:40:20.130970 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.62:35827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":77,"threads_started":1}
I20260929 18:40:20.317670 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 28595
I20260929 18:40:20.323738 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 28762
I20260929 18:40:20.328764 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 28896
I20260929 18:40:20.329135 29081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:20.328749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:35827 (local address 127.13.100.3:53575)
0929 18:40:20.328988 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0929 18:40:20.329040 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.62:35827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":88,"threads_started":1}
2026-09-29T18:40:20Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19359 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:40:20 dist-test-slave-j1xs krb5kdc[29084](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:40:20 dist-test-slave-j1xs krb5kdc[29084](info): set up 2 sockets
Sep 29 18:40:20 dist-test-slave-j1xs krb5kdc[29084](info): commencing operation
krb5kdc: starting...
W20260929 18:40:21.382658 13712 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/krb5kdc/test-user.keytab.
Sep 29 18:40:21 dist-test-slave-j1xs krb5kdc[29084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707221, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:40:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:40:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.100.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.100.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.100.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.62.keytab.
Entry for principal kudu/127.13.100.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.62.keytab.
I20260929 18:40:21.507007 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.100.62:34215
--webserver_interface=127.13.100.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.62.keytab
--principal=kudu/127.13.100.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.100.20:44697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.100.62:34215
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:40:21.615510 29097 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:40:21.615751 29097 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:40:21.615789 29097 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:40:21.619835 29097 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 18:40:21.619889 29097 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 18:40:21.619905 29097 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 18:40:21.619920 29097 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:40:21.619939 29097 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:40:21.619982 29097 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:40:21.620019 29097 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:40:21.620048 29097 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:40:21.625248 29097 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.100.20:44697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.13.100.62:34215
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.62.keytab
--principal=kudu/127.13.100.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.100.62:34215
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.100.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29097
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:40:21.626276 29097 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:40:21.627341 29097 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:40:21.636047 29102 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:40:21.636129 29103 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:40:21.636113 29105 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:40:21.636205 29097 server_base.cc:1034] running on GCE node
I20260929 18:40:21.636881 29097 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:40:21.637872 29097 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:40:21.639120 29097 hybrid_clock.cc:648] HybridClock initialized: now 1790707221639098 us; error 40 us; skew 500 ppm
Sep 29 18:40:21 dist-test-slave-j1xs krb5kdc[29084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707221, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:40:21.642064 29097 init.cc:377] Logged in from keytab as kudu/127.13.100.62@KRBTEST.COM (short username kudu)
I20260929 18:40:21.643031 29097 webserver.cc:460] Webserver started at http://127.13.100.62:42543/ using document root <none> and password file <none>
I20260929 18:40:21.643652 29097 fs_manager.cc:359] Metadata directory not provided
I20260929 18:40:21.643749 29097 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:40:21.643971 29097 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:40:21.646214 29097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/master-0/data/instance:
uuid: "55348a079a5f4d1795e48eda404bb105"
format_stamp: "Formatted at 2026-09-29 18:40:21 on dist-test-slave-j1xs"
server_key: "3028baaa3654086d63191a66f22284da"
server_key_iv: "f057f747b98d8b08403bbef039e93785"
server_key_version: "encryptionkey@0"
I20260929 18:40:21.646795 29097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/master-0/wal/instance:
uuid: "55348a079a5f4d1795e48eda404bb105"
format_stamp: "Formatted at 2026-09-29 18:40:21 on dist-test-slave-j1xs"
server_key: "3028baaa3654086d63191a66f22284da"
server_key_iv: "f057f747b98d8b08403bbef039e93785"
server_key_version: "encryptionkey@0"
I20260929 18:40:21.650609 29097 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 18:40:21.652987 29112 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:40:21.654258 29097 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260929 18:40:21.654428 29097 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/master-0/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/master-0/wal
uuid: "55348a079a5f4d1795e48eda404bb105"
format_stamp: "Formatted at 2026-09-29 18:40:21 on dist-test-slave-j1xs"
server_key: "3028baaa3654086d63191a66f22284da"
server_key_iv: "f057f747b98d8b08403bbef039e93785"
server_key_version: "encryptionkey@0"
I20260929 18:40:21.654541 29097 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:40:21.671392 29097 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:40:21.674896 29097 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:40:21.675112 29097 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:40:21.682109 29097 rpc_server.cc:307] RPC server started. Bound to: 127.13.100.62:34215
I20260929 18:40:21.682420 29164 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.100.62:34215 every 8 connection(s)
I20260929 18:40:21.683148 29097 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
I20260929 18:40:21.683527 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 29097
I20260929 18:40:21.683848 13712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/master-0/wal/instance
I20260929 18:40:21.684182 13712 external_mini_cluster.cc:1428] Setting key 1a0290801c7e22474933304cd808aef0
I20260929 18:40:21.686626 29165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:40:21.692483 29165 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55348a079a5f4d1795e48eda404bb105: Bootstrap starting.
Sep 29 18:40:21 dist-test-slave-j1xs krb5kdc[29084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707221, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.100.62@KRBTEST.COM
I20260929 18:40:21.695008 29165 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 55348a079a5f4d1795e48eda404bb105: Neither blocks nor log segments found. Creating new log.
I20260929 18:40:21.695796 29165 log.cc:824] T 00000000000000000000000000000000 P 55348a079a5f4d1795e48eda404bb105: Log is configured to *not* fsync() on all Append() calls
I20260929 18:40:21.697779 29165 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55348a079a5f4d1795e48eda404bb105: No bootstrap required, opened a new log
I20260929 18:40:21.699047 29168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:21.685790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52072 (local address 127.13.100.62:34215)
0929 18:40:21.686278 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0929 18:40:21.686296 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 18:40:21.686330 (+ 34us) server_negotiation.cc:407] Connection header received
0929 18:40:21.687348 (+ 1018us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:40:21.687369 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:40:21.687707 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:40:21.688092 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:40:21.688899 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:21.689879 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:21.690807 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:21.691274 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:21.691446 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:40:21.694248 (+ 2802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:40:21.694253 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:40:21.694266 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:40:21.694300 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:40:21.696165 (+ 1865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:40:21.696617 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:40:21.696622 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:40:21.696626 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:40:21.696733 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:40:21.697020 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:40:21.697025 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:40:21.697027 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:40:21.697527 (+ 500us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:40:21.697766 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0929 18:40:21.698052 (+ 286us) server_negotiation.cc:299] Negotiation successful
0929 18:40:21.698350 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":159,"threads_started":1}
I20260929 18:40:21.702239 29165 raft_consensus.cc:348] T 00000000000000000000000000000000 P 55348a079a5f4d1795e48eda404bb105 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55348a079a5f4d1795e48eda404bb105" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 34215 } }
I20260929 18:40:21.702502 29165 raft_consensus.cc:374] T 00000000000000000000000000000000 P 55348a079a5f4d1795e48eda404bb105 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:40:21.702593 29165 raft_consensus.cc:729] T 00000000000000000000000000000000 P 55348a079a5f4d1795e48eda404bb105 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55348a079a5f4d1795e48eda404bb105, State: Initialized, Role: FOLLOWER
I20260929 18:40:21.703047 29165 consensus_queue.cc:260] T 00000000000000000000000000000000 P 55348a079a5f4d1795e48eda404bb105 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55348a079a5f4d1795e48eda404bb105" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 34215 } }
I20260929 18:40:21.703202 29165 raft_consensus.cc:388] T 00000000000000000000000000000000 P 55348a079a5f4d1795e48eda404bb105 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:40:21.703295 29165 raft_consensus.cc:482] T 00000000000000000000000000000000 P 55348a079a5f4d1795e48eda404bb105 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:40:21.703423 29165 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 55348a079a5f4d1795e48eda404bb105 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:40:21.704522 29165 raft_consensus.cc:504] T 00000000000000000000000000000000 P 55348a079a5f4d1795e48eda404bb105 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55348a079a5f4d1795e48eda404bb105" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 34215 } }
I20260929 18:40:21.704875 29165 leader_election.cc:302] T 00000000000000000000000000000000 P 55348a079a5f4d1795e48eda404bb105 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 55348a079a5f4d1795e48eda404bb105; no voters:
I20260929 18:40:21.705194 29165 leader_election.cc:288] T 00000000000000000000000000000000 P 55348a079a5f4d1795e48eda404bb105 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:40:21.705350 29170 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 55348a079a5f4d1795e48eda404bb105 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:40:21.705860 29170 raft_consensus.cc:686] T 00000000000000000000000000000000 P 55348a079a5f4d1795e48eda404bb105 [term 1 LEADER]: Becoming Leader. State: Replica: 55348a079a5f4d1795e48eda404bb105, State: Running, Role: LEADER
I20260929 18:40:21.706347 29170 consensus_queue.cc:237] T 00000000000000000000000000000000 P 55348a079a5f4d1795e48eda404bb105 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55348a079a5f4d1795e48eda404bb105" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 34215 } }
I20260929 18:40:21.706584 29165 sys_catalog.cc:564] T 00000000000000000000000000000000 P 55348a079a5f4d1795e48eda404bb105 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:40:21.708338 29172 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55348a079a5f4d1795e48eda404bb105 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 55348a079a5f4d1795e48eda404bb105. Latest consensus state: current_term: 1 leader_uuid: "55348a079a5f4d1795e48eda404bb105" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55348a079a5f4d1795e48eda404bb105" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 34215 } } }
I20260929 18:40:21.708316 29171 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55348a079a5f4d1795e48eda404bb105 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "55348a079a5f4d1795e48eda404bb105" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55348a079a5f4d1795e48eda404bb105" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 34215 } } }
I20260929 18:40:21.708494 29172 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55348a079a5f4d1795e48eda404bb105 [sys.catalog]: This master's current role is: LEADER
I20260929 18:40:21.708498 29171 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55348a079a5f4d1795e48eda404bb105 [sys.catalog]: This master's current role is: LEADER
I20260929 18:40:21.709201 29180 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:40:21.713186 29180 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:40:21.718879 29180 catalog_manager.cc:1347] Generated new cluster ID: e62d58bc40c54a52be38c7f6f3340271
I20260929 18:40:21.718952 29180 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:40:21.745474 29180 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:40:21.746874 29180 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:40:21.769213 29180 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 55348a079a5f4d1795e48eda404bb105: Generated new TSK 0
I20260929 18:40:21.770162 29180 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.100.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.100.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.100.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.1.keytab.
Entry for principal kudu/127.13.100.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.1.keytab.
I20260929 18:40:21.807253 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.100.1:0
--local_ip_for_outbound_sockets=127.13.100.1
--webserver_interface=127.13.100.1
--webserver_port=0
--tserver_master_addrs=127.13.100.62:34215
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.1.keytab
--principal=kudu/127.13.100.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.100.20:44697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:40:21.919836 29191 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:40:21.920090 29191 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:40:21.920135 29191 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:40:21.924060 29191 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:40:21.924114 29191 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:40:21.924175 29191 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.100.1
I20260929 18:40:21.929275 29191 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.100.20:44697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.1.keytab
--principal=kudu/127.13.100.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.100.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.100.1
--webserver_port=0
--tserver_master_addrs=127.13.100.62:34215
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29191
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.100.1
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:40:21.930200 29191 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:40:21.931134 29191 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:40:21.942714 29196 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:40:21.942814 29191 server_base.cc:1034] running on GCE node
W20260929 18:40:21.942714 29197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:40:21.942721 29199 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:40:21.943383 29191 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:40:21.943969 29191 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:40:21.945148 29191 hybrid_clock.cc:648] HybridClock initialized: now 1790707221945127 us; error 44 us; skew 500 ppm
Sep 29 18:40:21 dist-test-slave-j1xs krb5kdc[29084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707221, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:40:21.947817 29191 init.cc:377] Logged in from keytab as kudu/127.13.100.1@KRBTEST.COM (short username kudu)
I20260929 18:40:21.948625 29191 webserver.cc:460] Webserver started at http://127.13.100.1:38303/ using document root <none> and password file <none>
I20260929 18:40:21.949108 29191 fs_manager.cc:359] Metadata directory not provided
I20260929 18:40:21.949164 29191 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:40:21.949317 29191 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:40:21.951282 29191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/ts-0/data/instance:
uuid: "dfa07a7f1fec49beb458bd72df7589e7"
format_stamp: "Formatted at 2026-09-29 18:40:21 on dist-test-slave-j1xs"
server_key: "678290f32b6284b80e3321b76a46581c"
server_key_iv: "546083d1f5bf912f0549e28f3e4e8826"
server_key_version: "encryptionkey@0"
I20260929 18:40:21.951776 29191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/ts-0/wal/instance:
uuid: "dfa07a7f1fec49beb458bd72df7589e7"
format_stamp: "Formatted at 2026-09-29 18:40:21 on dist-test-slave-j1xs"
server_key: "678290f32b6284b80e3321b76a46581c"
server_key_iv: "546083d1f5bf912f0549e28f3e4e8826"
server_key_version: "encryptionkey@0"
I20260929 18:40:21.955284 29191 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 18:40:21.957526 29206 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:40:21.958921 29191 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:40:21.959012 29191 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/ts-0/wal
uuid: "dfa07a7f1fec49beb458bd72df7589e7"
format_stamp: "Formatted at 2026-09-29 18:40:21 on dist-test-slave-j1xs"
server_key: "678290f32b6284b80e3321b76a46581c"
server_key_iv: "546083d1f5bf912f0549e28f3e4e8826"
server_key_version: "encryptionkey@0"
I20260929 18:40:21.959092 29191 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:40:21.985074 29191 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:40:21.988107 29191 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:40:21.988256 29191 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:40:21.988776 29191 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:40:21.989517 29191 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:40:21.989563 29191 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:40:21.989603 29191 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:40:21.989619 29191 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:40:21.998646 29191 rpc_server.cc:307] RPC server started. Bound to: 127.13.100.1:36845
I20260929 18:40:21.998684 29319 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.100.1:36845 every 8 connection(s)
I20260929 18:40:21.999487 29191 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/ts-0/data/info.pb
I20260929 18:40:22.004402 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 29191
I20260929 18:40:22.004542 13712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/ts-0/wal/instance
I20260929 18:40:22.004844 13712 external_mini_cluster.cc:1428] Setting key 4da8bad90148ae9224190b9d406c7236
Sep 29 18:40:22 dist-test-slave-j1xs krb5kdc[29084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707221, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.1@KRBTEST.COM for kudu/127.13.100.62@KRBTEST.COM
I20260929 18:40:22.013978 29168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:22.001273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.1:60229 (local address 127.13.100.62:34215)
0929 18:40:22.001426 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0929 18:40:22.001430 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:40:22.002333 (+ 903us) server_negotiation.cc:407] Connection header received
0929 18:40:22.003174 (+ 841us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:40:22.003176 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:40:22.003219 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:40:22.003291 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:40:22.004475 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:22.005242 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:22.006642 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:22.007047 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:22.007170 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:40:22.009995 (+ 2825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:40:22.010001 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:40:22.010008 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:40:22.010041 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:40:22.011880 (+ 1839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:40:22.012374 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:40:22.012379 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:40:22.012382 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:40:22.012476 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:40:22.012797 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:40:22.012801 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:40:22.012803 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:40:22.013046 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:40:22.013151 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 18:40:22.013630 (+ 479us) server_negotiation.cc:299] Negotiation successful
0929 18:40:22.013762 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 18:40:22.014670 29322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:22.001472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:34215 (local address 127.13.100.1:60229)
0929 18:40:22.002183 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0929 18:40:22.002225 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 18:40:22.002996 (+ 771us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:40:22.003450 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:40:22.003456 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:40:22.003824 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:40:22.004290 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:40:22.004304 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:40:22.005397 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:40:22.005402 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:40:22.006523 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:40:22.006533 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:40:22.007196 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:40:22.007199 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:40:22.007387 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:40:22.008108 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:40:22.008133 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:40:22.009806 (+ 1673us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:40:22.011999 (+ 2193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:40:22.012003 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:40:22.012013 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:40:22.012256 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:40:22.012572 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:40:22.012574 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:40:22.012575 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:40:22.012693 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:40:22.013150 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:40:22.013155 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:40:22.013365 (+ 210us) client_negotiation.cc:769] Sending connection context
0929 18:40:22.013520 (+ 155us) client_negotiation.cc:240] Negotiation successful
0929 18:40:22.013755 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":110,"threads_started":1}
WARNING: no policy specified for kudu/127.13.100.2@KRBTEST.COM; defaulting to no policy
I20260929 18:40:22.015863 29320 heartbeater.cc:347] Connected to a master server at 127.13.100.62:34215
I20260929 18:40:22.016103 29320 heartbeater.cc:464] Registering TS with master...
I20260929 18:40:22.016562 29320 heartbeater.cc:511] Master 127.13.100.62:34215 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.100.2@KRBTEST.COM" created.
I20260929 18:40:22.018177 29129 ts_manager.cc:194] Registered new tserver with Master: dfa07a7f1fec49beb458bd72df7589e7 (127.13.100.1:36845)
I20260929 18:40:22.019268 29129 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.100.1@KRBTEST.COM'} at 127.13.100.1:60229
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.100.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.2.keytab.
Entry for principal kudu/127.13.100.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.2.keytab.
I20260929 18:40:22.037277 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.100.2:0
--local_ip_for_outbound_sockets=127.13.100.2
--webserver_interface=127.13.100.2
--webserver_port=0
--tserver_master_addrs=127.13.100.62:34215
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.2.keytab
--principal=kudu/127.13.100.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.100.20:44697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:40:22.146867 29325 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:40:22.147121 29325 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:40:22.147164 29325 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:40:22.151229 29325 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:40:22.151290 29325 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:40:22.151377 29325 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.100.2
I20260929 18:40:22.156571 29325 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.100.20:44697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.2.keytab
--principal=kudu/127.13.100.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.100.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.100.2
--webserver_port=0
--tserver_master_addrs=127.13.100.62:34215
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29325
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.100.2
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:40:22.157581 29325 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:40:22.158615 29325 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:40:22.168272 29333 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:40:22.168494 29325 server_base.cc:1034] running on GCE node
W20260929 18:40:22.168278 29331 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:40:22.168272 29330 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:40:22.169139 29325 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:40:22.169737 29325 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:40:22.170907 29325 hybrid_clock.cc:648] HybridClock initialized: now 1790707222170889 us; error 44 us; skew 500 ppm
Sep 29 18:40:22 dist-test-slave-j1xs krb5kdc[29084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707222, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:40:22.173643 29325 init.cc:377] Logged in from keytab as kudu/127.13.100.2@KRBTEST.COM (short username kudu)
I20260929 18:40:22.174578 29325 webserver.cc:460] Webserver started at http://127.13.100.2:36695/ using document root <none> and password file <none>
I20260929 18:40:22.175144 29325 fs_manager.cc:359] Metadata directory not provided
I20260929 18:40:22.175235 29325 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:40:22.175470 29325 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:40:22.177606 29325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/ts-1/data/instance:
uuid: "248d7dfbf266486b99f1abfc09afcb8e"
format_stamp: "Formatted at 2026-09-29 18:40:22 on dist-test-slave-j1xs"
server_key: "dcf8d37d7b3add7c0354687c0d3bfd2e"
server_key_iv: "56e823f10c16a4a9bb098bf07a198f47"
server_key_version: "encryptionkey@0"
I20260929 18:40:22.178190 29325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/ts-1/wal/instance:
uuid: "248d7dfbf266486b99f1abfc09afcb8e"
format_stamp: "Formatted at 2026-09-29 18:40:22 on dist-test-slave-j1xs"
server_key: "dcf8d37d7b3add7c0354687c0d3bfd2e"
server_key_iv: "56e823f10c16a4a9bb098bf07a198f47"
server_key_version: "encryptionkey@0"
I20260929 18:40:22.181767 29325 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 18:40:22.184226 29340 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:40:22.185487 29325 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:40:22.185622 29325 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/ts-1/wal
uuid: "248d7dfbf266486b99f1abfc09afcb8e"
format_stamp: "Formatted at 2026-09-29 18:40:22 on dist-test-slave-j1xs"
server_key: "dcf8d37d7b3add7c0354687c0d3bfd2e"
server_key_iv: "56e823f10c16a4a9bb098bf07a198f47"
server_key_version: "encryptionkey@0"
I20260929 18:40:22.185741 29325 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:40:22.196271 29325 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:40:22.199802 29325 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:40:22.200023 29325 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:40:22.200639 29325 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:40:22.201542 29325 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:40:22.201613 29325 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:40:22.201681 29325 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:40:22.201730 29325 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:40:22.211299 29325 rpc_server.cc:307] RPC server started. Bound to: 127.13.100.2:36851
I20260929 18:40:22.211318 29453 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.100.2:36851 every 8 connection(s)
I20260929 18:40:22.212205 29325 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/ts-1/data/info.pb
I20260929 18:40:22.213160 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 29325
I20260929 18:40:22.213296 13712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/ts-1/wal/instance
I20260929 18:40:22.213619 13712 external_mini_cluster.cc:1428] Setting key f6d2f9575110f756297e42562711d704
WARNING: no policy specified for kudu/127.13.100.3@KRBTEST.COM; defaulting to no policy
Sep 29 18:40:22 dist-test-slave-j1xs krb5kdc[29084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707222, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.2@KRBTEST.COM for kudu/127.13.100.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.100.3@KRBTEST.COM" created.
I20260929 18:40:22.228865 29168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:22.214269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.2:44467 (local address 127.13.100.62:34215)
0929 18:40:22.214425 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0929 18:40:22.214429 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:40:22.216207 (+ 1778us) server_negotiation.cc:407] Connection header received
0929 18:40:22.217040 (+ 833us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:40:22.217046 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:40:22.217109 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:40:22.217208 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:40:22.218569 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:22.219077 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:22.220945 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:22.221173 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:22.221313 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:40:22.224357 (+ 3044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:40:22.224360 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:40:22.224363 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:40:22.224390 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:40:22.226216 (+ 1826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:40:22.226959 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:40:22.226963 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:40:22.226965 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:40:22.227061 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:40:22.227380 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:40:22.227386 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:40:22.227391 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:40:22.227666 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:40:22.227770 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 18:40:22.228562 (+ 792us) server_negotiation.cc:299] Negotiation successful
0929 18:40:22.228694 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 18:40:22.229773 29456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:22.214807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:34215 (local address 127.13.100.2:44467)
0929 18:40:22.216060 (+ 1253us) negotiation.cc:107] Waiting for socket to connect
0929 18:40:22.216101 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 18:40:22.216838 (+ 737us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:40:22.217352 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:40:22.217359 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:40:22.217720 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:40:22.218407 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:40:22.218428 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:40:22.219202 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:40:22.219208 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:40:22.220795 (+ 1587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:40:22.220809 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:40:22.221369 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:40:22.221376 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:40:22.221608 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:40:22.222293 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:40:22.222317 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:40:22.224171 (+ 1854us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:40:22.226322 (+ 2151us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:40:22.226345 (+ 23us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:40:22.226365 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:40:22.226818 (+ 453us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:40:22.227155 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:40:22.227157 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:40:22.227158 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:40:22.227288 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:40:22.228016 (+ 728us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:40:22.228024 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:40:22.228307 (+ 283us) client_negotiation.cc:769] Sending connection context
0929 18:40:22.228521 (+ 214us) client_negotiation.cc:240] Negotiation successful
0929 18:40:22.228848 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1060,"thread_start_us":169,"threads_started":1}
I20260929 18:40:22.231215 29454 heartbeater.cc:347] Connected to a master server at 127.13.100.62:34215
I20260929 18:40:22.231477 29454 heartbeater.cc:464] Registering TS with master...
I20260929 18:40:22.231983 29454 heartbeater.cc:511] Master 127.13.100.62:34215 requested a full tablet report, sending...
I20260929 18:40:22.233124 29129 ts_manager.cc:194] Registered new tserver with Master: 248d7dfbf266486b99f1abfc09afcb8e (127.13.100.2:36851)
I20260929 18:40:22.233994 29129 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.100.2@KRBTEST.COM'} at 127.13.100.2:44467
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.100.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.3.keytab.
Entry for principal kudu/127.13.100.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.3.keytab.
I20260929 18:40:22.242949 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.100.3:0
--local_ip_for_outbound_sockets=127.13.100.3
--webserver_interface=127.13.100.3
--webserver_port=0
--tserver_master_addrs=127.13.100.62:34215
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.3.keytab
--principal=kudu/127.13.100.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.100.20:44697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:40:22.351195 29459 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:40:22.351451 29459 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:40:22.351493 29459 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:40:22.355335 29459 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:40:22.355384 29459 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:40:22.355441 29459 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.100.3
I20260929 18:40:22.360635 29459 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.100.20:44697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.3.keytab
--principal=kudu/127.13.100.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.100.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.100.3
--webserver_port=0
--tserver_master_addrs=127.13.100.62:34215
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29459
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.100.3
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:40:22.361567 29459 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:40:22.362577 29459 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:40:22.371584 29467 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:40:22.371644 29465 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:40:22.371589 29464 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:40:22.371752 29459 server_base.cc:1034] running on GCE node
I20260929 18:40:22.372225 29459 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:40:22.372826 29459 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:40:22.373960 29459 hybrid_clock.cc:648] HybridClock initialized: now 1790707222373927 us; error 48 us; skew 500 ppm
Sep 29 18:40:22 dist-test-slave-j1xs krb5kdc[29084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707222, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:40:22.376621 29459 init.cc:377] Logged in from keytab as kudu/127.13.100.3@KRBTEST.COM (short username kudu)
I20260929 18:40:22.377406 29459 webserver.cc:460] Webserver started at http://127.13.100.3:34335/ using document root <none> and password file <none>
I20260929 18:40:22.378005 29459 fs_manager.cc:359] Metadata directory not provided
I20260929 18:40:22.378070 29459 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:40:22.378229 29459 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:40:22.380172 29459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/ts-2/data/instance:
uuid: "0f8af5f5a4024455869ea7a896d2364f"
format_stamp: "Formatted at 2026-09-29 18:40:22 on dist-test-slave-j1xs"
server_key: "522c02267cf80687cb19146bab7805ee"
server_key_iv: "b781fd704777451fefe12e45a4893748"
server_key_version: "encryptionkey@0"
I20260929 18:40:22.380710 29459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/ts-2/wal/instance:
uuid: "0f8af5f5a4024455869ea7a896d2364f"
format_stamp: "Formatted at 2026-09-29 18:40:22 on dist-test-slave-j1xs"
server_key: "522c02267cf80687cb19146bab7805ee"
server_key_iv: "b781fd704777451fefe12e45a4893748"
server_key_version: "encryptionkey@0"
I20260929 18:40:22.384341 29459 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 18:40:22.386762 29474 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:40:22.388000 29459 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:40:22.388120 29459 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/ts-2/wal
uuid: "0f8af5f5a4024455869ea7a896d2364f"
format_stamp: "Formatted at 2026-09-29 18:40:22 on dist-test-slave-j1xs"
server_key: "522c02267cf80687cb19146bab7805ee"
server_key_iv: "b781fd704777451fefe12e45a4893748"
server_key_version: "encryptionkey@0"
I20260929 18:40:22.388226 29459 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:40:22.411923 29459 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:40:22.415345 29459 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:40:22.415565 29459 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:40:22.416164 29459 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:40:22.416972 29459 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:40:22.417032 29459 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:40:22.417101 29459 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:40:22.417140 29459 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:40:22.426103 29459 rpc_server.cc:307] RPC server started. Bound to: 127.13.100.3:46121
I20260929 18:40:22.426127 29587 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.100.3:46121 every 8 connection(s)
I20260929 18:40:22.426949 29459 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/ts-2/data/info.pb
I20260929 18:40:22.429386 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 29459
I20260929 18:40:22.429526 13712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/ts-2/wal/instance
I20260929 18:40:22.429829 13712 external_mini_cluster.cc:1428] Setting key 7806280c56d22cade1333e4181522fc4
Sep 29 18:40:22 dist-test-slave-j1xs krb5kdc[29084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707222, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.3@KRBTEST.COM for kudu/127.13.100.62@KRBTEST.COM
I20260929 18:40:22.439741 29168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:22.428675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.3:48391 (local address 127.13.100.62:34215)
0929 18:40:22.428808 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0929 18:40:22.428811 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:40:22.430022 (+ 1211us) server_negotiation.cc:407] Connection header received
0929 18:40:22.431039 (+ 1017us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:40:22.431042 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:40:22.431097 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:40:22.431201 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:40:22.432290 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:22.432832 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:22.433986 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:22.434199 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:22.434309 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:40:22.436566 (+ 2257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:40:22.436572 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:40:22.436578 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:40:22.436602 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:40:22.437799 (+ 1197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:40:22.438298 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:40:22.438303 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:40:22.438305 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:40:22.438380 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:40:22.438712 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:40:22.438715 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:40:22.438717 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:40:22.438886 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:40:22.438990 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 18:40:22.439456 (+ 466us) server_negotiation.cc:299] Negotiation successful
0929 18:40:22.439578 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 18:40:22.440441 29590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:22.429081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:34215 (local address 127.13.100.3:48391)
0929 18:40:22.429833 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0929 18:40:22.429871 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 18:40:22.430849 (+ 978us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:40:22.431364 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:40:22.431371 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:40:22.431707 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:40:22.432154 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:40:22.432167 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:40:22.432951 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:40:22.432955 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:40:22.433816 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:40:22.433825 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:40:22.434343 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:40:22.434347 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:40:22.434519 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:40:22.435052 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:40:22.435068 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:40:22.436414 (+ 1346us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:40:22.437929 (+ 1515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:40:22.437935 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:40:22.437949 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:40:22.438194 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:40:22.438472 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:40:22.438476 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:40:22.438480 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:40:22.438611 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:40:22.439006 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:40:22.439013 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:40:22.439231 (+ 218us) client_negotiation.cc:769] Sending connection context
0929 18:40:22.439405 (+ 174us) client_negotiation.cc:240] Negotiation successful
0929 18:40:22.439696 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"thread_start_us":168,"threads_started":1}
I20260929 18:40:22.441466 29588 heartbeater.cc:347] Connected to a master server at 127.13.100.62:34215
I20260929 18:40:22.441670 29588 heartbeater.cc:464] Registering TS with master...
I20260929 18:40:22.442130 29588 heartbeater.cc:511] Master 127.13.100.62:34215 requested a full tablet report, sending...
I20260929 18:40:22.443140 29129 ts_manager.cc:194] Registered new tserver with Master: 0f8af5f5a4024455869ea7a896d2364f (127.13.100.3:46121)
I20260929 18:40:22.443758 29129 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.100.3@KRBTEST.COM'} at 127.13.100.3:48391
I20260929 18:40:22.455206 13712 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 18:40:22.465040 29168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:22.457110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52076 (local address 127.13.100.62:34215)
0929 18:40:22.457228 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0929 18:40:22.457231 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:40:22.457463 (+ 232us) server_negotiation.cc:407] Connection header received
0929 18:40:22.457562 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:40:22.457564 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:40:22.457610 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:40:22.457719 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:40:22.458497 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:22.459053 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:22.459884 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:22.460093 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:22.460180 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:40:22.461586 (+ 1406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:40:22.461589 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:40:22.461591 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:40:22.461613 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:40:22.463194 (+ 1581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:40:22.463717 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:40:22.463720 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:40:22.463721 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:40:22.463795 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:40:22.464092 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:40:22.464094 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:40:22.464095 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:40:22.464323 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:40:22.464424 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 18:40:22.464801 (+ 377us) server_negotiation.cc:299] Negotiation successful
0929 18:40:22.464914 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 18:40:22.467988 29129 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52076:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:40:22.471016 29129 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:40:22.485924 29599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:22.479503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.3:46121 (local address 127.0.0.1:34970)
0929 18:40:22.479862 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0929 18:40:22.479917 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 18:40:22.480059 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:40:22.481101 (+ 1042us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:40:22.481107 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:40:22.481143 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:40:22.481318 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:40:22.481328 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:40:22.482958 (+ 1630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:40:22.482961 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:40:22.484221 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:40:22.484231 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:40:22.485508 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:40:22.485514 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:40:22.485609 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:40:22.485628 (+ 19us) client_negotiation.cc:769] Sending connection context
0929 18:40:22.485693 (+ 65us) client_negotiation.cc:240] Negotiation successful
0929 18:40:22.485780 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"mutex_wait_us":103,"spinlock_wait_cycles":6528,"thread_start_us":114,"threads_started":1}
I20260929 18:40:22.486018 29604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:22.480489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58122 (local address 127.13.100.2:36851)
0929 18:40:22.481037 (+ 548us) server_negotiation.cc:206] Beginning negotiation
0929 18:40:22.481049 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 18:40:22.481071 (+ 22us) server_negotiation.cc:407] Connection header received
0929 18:40:22.481140 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:40:22.481145 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:40:22.481325 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:40:22.481501 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:40:22.481747 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:22.482548 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:22.484578 (+ 2030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:22.485138 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:22.485261 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:40:22.485426 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0929 18:40:22.485707 (+ 281us) server_negotiation.cc:299] Negotiation successful
0929 18:40:22.485825 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":445,"thread_start_us":101,"threads_started":1}
I20260929 18:40:22.486090 29601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:22.479686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.1:36845 (local address 127.0.0.1:51414)
0929 18:40:22.480312 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0929 18:40:22.480334 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 18:40:22.480413 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:40:22.481101 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:40:22.481107 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:40:22.481142 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:40:22.481461 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:40:22.481467 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:40:22.483009 (+ 1542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:40:22.483012 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:40:22.484169 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:40:22.484178 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:40:22.485588 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:40:22.485591 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:40:22.485695 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:40:22.485742 (+ 47us) client_negotiation.cc:769] Sending connection context
0929 18:40:22.485810 (+ 68us) client_negotiation.cc:240] Negotiation successful
0929 18:40:22.485873 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"mutex_wait_us":278,"spinlock_wait_cycles":5760,"thread_start_us":85,"threads_started":1}
I20260929 18:40:22.485842 29603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:22.480253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.2:36851 (local address 127.0.0.1:58122)
0929 18:40:22.480543 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0929 18:40:22.480558 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 18:40:22.480650 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:40:22.481496 (+ 846us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:40:22.481499 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:40:22.481515 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:40:22.481638 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:40:22.481644 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:40:22.482676 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:40:22.482681 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:40:22.484470 (+ 1789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:40:22.484477 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:40:22.485261 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:40:22.485265 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:40:22.485382 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:40:22.485420 (+ 38us) client_negotiation.cc:769] Sending connection context
0929 18:40:22.485528 (+ 108us) client_negotiation.cc:240] Negotiation successful
0929 18:40:22.485670 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"mutex_wait_us":82,"thread_start_us":85,"threads_started":1}
I20260929 18:40:22.486231 29602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:22.480172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51414 (local address 127.13.100.1:36845)
0929 18:40:22.480623 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0929 18:40:22.480637 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 18:40:22.480656 (+ 19us) server_negotiation.cc:407] Connection header received
0929 18:40:22.480719 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:40:22.480722 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:40:22.480925 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:40:22.481212 (+ 287us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:40:22.481581 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:22.482820 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:22.484289 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:22.485254 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:22.485364 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:40:22.485524 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0929 18:40:22.485969 (+ 445us) server_negotiation.cc:299] Negotiation successful
0929 18:40:22.486076 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":90,"threads_started":1}
I20260929 18:40:22.487161 29600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:22.479636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34970 (local address 127.13.100.3:46121)
0929 18:40:22.480603 (+ 967us) server_negotiation.cc:206] Beginning negotiation
0929 18:40:22.480615 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 18:40:22.480639 (+ 24us) server_negotiation.cc:407] Connection header received
0929 18:40:22.480717 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:40:22.480722 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:40:22.480925 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:40:22.481475 (+ 550us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:40:22.481641 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:22.482820 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:22.484482 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:22.485367 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:22.485498 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:40:22.485618 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0929 18:40:22.486918 (+ 1300us) server_negotiation.cc:299] Negotiation successful
0929 18:40:22.487019 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":866,"thread_start_us":106,"threads_started":1}
I20260929 18:40:22.489074 29388 tablet_service.cc:1467] Processing CreateTablet for tablet 0b1d9029f71d4111bd46385487da6cad (DEFAULT_TABLE table=test-table [id=6bf868ed94424da0a4856b77104f7a47]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:40:22.489081 29254 tablet_service.cc:1467] Processing CreateTablet for tablet 0b1d9029f71d4111bd46385487da6cad (DEFAULT_TABLE table=test-table [id=6bf868ed94424da0a4856b77104f7a47]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:40:22.489730 29522 tablet_service.cc:1467] Processing CreateTablet for tablet 0b1d9029f71d4111bd46385487da6cad (DEFAULT_TABLE table=test-table [id=6bf868ed94424da0a4856b77104f7a47]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:40:22.490130 29388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b1d9029f71d4111bd46385487da6cad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:40:22.490142 29254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b1d9029f71d4111bd46385487da6cad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:40:22.490664 29522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b1d9029f71d4111bd46385487da6cad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:40:22.495903 29605 tablet_bootstrap.cc:492] T 0b1d9029f71d4111bd46385487da6cad P 248d7dfbf266486b99f1abfc09afcb8e: Bootstrap starting.
I20260929 18:40:22.497792 29606 tablet_bootstrap.cc:492] T 0b1d9029f71d4111bd46385487da6cad P dfa07a7f1fec49beb458bd72df7589e7: Bootstrap starting.
I20260929 18:40:22.498373 29605 tablet_bootstrap.cc:654] T 0b1d9029f71d4111bd46385487da6cad P 248d7dfbf266486b99f1abfc09afcb8e: Neither blocks nor log segments found. Creating new log.
I20260929 18:40:22.498764 29607 tablet_bootstrap.cc:492] T 0b1d9029f71d4111bd46385487da6cad P 0f8af5f5a4024455869ea7a896d2364f: Bootstrap starting.
I20260929 18:40:22.499279 29605 log.cc:824] T 0b1d9029f71d4111bd46385487da6cad P 248d7dfbf266486b99f1abfc09afcb8e: Log is configured to *not* fsync() on all Append() calls
I20260929 18:40:22.499974 29606 tablet_bootstrap.cc:654] T 0b1d9029f71d4111bd46385487da6cad P dfa07a7f1fec49beb458bd72df7589e7: Neither blocks nor log segments found. Creating new log.
I20260929 18:40:22.500648 29607 tablet_bootstrap.cc:654] T 0b1d9029f71d4111bd46385487da6cad P 0f8af5f5a4024455869ea7a896d2364f: Neither blocks nor log segments found. Creating new log.
I20260929 18:40:22.501168 29606 log.cc:824] T 0b1d9029f71d4111bd46385487da6cad P dfa07a7f1fec49beb458bd72df7589e7: Log is configured to *not* fsync() on all Append() calls
I20260929 18:40:22.501426 29605 tablet_bootstrap.cc:492] T 0b1d9029f71d4111bd46385487da6cad P 248d7dfbf266486b99f1abfc09afcb8e: No bootstrap required, opened a new log
I20260929 18:40:22.501629 29605 ts_tablet_manager.cc:1397] T 0b1d9029f71d4111bd46385487da6cad P 248d7dfbf266486b99f1abfc09afcb8e: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260929 18:40:22.501693 29607 log.cc:824] T 0b1d9029f71d4111bd46385487da6cad P 0f8af5f5a4024455869ea7a896d2364f: Log is configured to *not* fsync() on all Append() calls
I20260929 18:40:22.503129 29606 tablet_bootstrap.cc:492] T 0b1d9029f71d4111bd46385487da6cad P dfa07a7f1fec49beb458bd72df7589e7: No bootstrap required, opened a new log
I20260929 18:40:22.503312 29606 ts_tablet_manager.cc:1397] T 0b1d9029f71d4111bd46385487da6cad P dfa07a7f1fec49beb458bd72df7589e7: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 18:40:22.503525 29607 tablet_bootstrap.cc:492] T 0b1d9029f71d4111bd46385487da6cad P 0f8af5f5a4024455869ea7a896d2364f: No bootstrap required, opened a new log
I20260929 18:40:22.503690 29607 ts_tablet_manager.cc:1397] T 0b1d9029f71d4111bd46385487da6cad P 0f8af5f5a4024455869ea7a896d2364f: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 18:40:22.505096 29605 raft_consensus.cc:348] T 0b1d9029f71d4111bd46385487da6cad P 248d7dfbf266486b99f1abfc09afcb8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f8af5f5a4024455869ea7a896d2364f" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 46121 } } peers { permanent_uuid: "dfa07a7f1fec49beb458bd72df7589e7" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 36845 } } peers { permanent_uuid: "248d7dfbf266486b99f1abfc09afcb8e" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 36851 } }
I20260929 18:40:22.505342 29605 raft_consensus.cc:374] T 0b1d9029f71d4111bd46385487da6cad P 248d7dfbf266486b99f1abfc09afcb8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:40:22.505425 29605 raft_consensus.cc:729] T 0b1d9029f71d4111bd46385487da6cad P 248d7dfbf266486b99f1abfc09afcb8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 248d7dfbf266486b99f1abfc09afcb8e, State: Initialized, Role: FOLLOWER
I20260929 18:40:22.505878 29605 consensus_queue.cc:260] T 0b1d9029f71d4111bd46385487da6cad P 248d7dfbf266486b99f1abfc09afcb8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f8af5f5a4024455869ea7a896d2364f" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 46121 } } peers { permanent_uuid: "dfa07a7f1fec49beb458bd72df7589e7" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 36845 } } peers { permanent_uuid: "248d7dfbf266486b99f1abfc09afcb8e" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 36851 } }
I20260929 18:40:22.506600 29454 heartbeater.cc:503] Master 127.13.100.62:34215 was elected leader, sending a full tablet report...
I20260929 18:40:22.506960 29605 ts_tablet_manager.cc:1428] T 0b1d9029f71d4111bd46385487da6cad P 248d7dfbf266486b99f1abfc09afcb8e: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260929 18:40:22.507263 29606 raft_consensus.cc:348] T 0b1d9029f71d4111bd46385487da6cad P dfa07a7f1fec49beb458bd72df7589e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f8af5f5a4024455869ea7a896d2364f" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 46121 } } peers { permanent_uuid: "dfa07a7f1fec49beb458bd72df7589e7" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 36845 } } peers { permanent_uuid: "248d7dfbf266486b99f1abfc09afcb8e" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 36851 } }
I20260929 18:40:22.507233 29607 raft_consensus.cc:348] T 0b1d9029f71d4111bd46385487da6cad P 0f8af5f5a4024455869ea7a896d2364f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f8af5f5a4024455869ea7a896d2364f" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 46121 } } peers { permanent_uuid: "dfa07a7f1fec49beb458bd72df7589e7" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 36845 } } peers { permanent_uuid: "248d7dfbf266486b99f1abfc09afcb8e" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 36851 } }
I20260929 18:40:22.507471 29607 raft_consensus.cc:374] T 0b1d9029f71d4111bd46385487da6cad P 0f8af5f5a4024455869ea7a896d2364f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:40:22.507483 29606 raft_consensus.cc:374] T 0b1d9029f71d4111bd46385487da6cad P dfa07a7f1fec49beb458bd72df7589e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:40:22.507529 29606 raft_consensus.cc:729] T 0b1d9029f71d4111bd46385487da6cad P dfa07a7f1fec49beb458bd72df7589e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfa07a7f1fec49beb458bd72df7589e7, State: Initialized, Role: FOLLOWER
I20260929 18:40:22.507532 29607 raft_consensus.cc:729] T 0b1d9029f71d4111bd46385487da6cad P 0f8af5f5a4024455869ea7a896d2364f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f8af5f5a4024455869ea7a896d2364f, State: Initialized, Role: FOLLOWER
I20260929 18:40:22.507977 29606 consensus_queue.cc:260] T 0b1d9029f71d4111bd46385487da6cad P dfa07a7f1fec49beb458bd72df7589e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f8af5f5a4024455869ea7a896d2364f" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 46121 } } peers { permanent_uuid: "dfa07a7f1fec49beb458bd72df7589e7" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 36845 } } peers { permanent_uuid: "248d7dfbf266486b99f1abfc09afcb8e" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 36851 } }
I20260929 18:40:22.507983 29607 consensus_queue.cc:260] T 0b1d9029f71d4111bd46385487da6cad P 0f8af5f5a4024455869ea7a896d2364f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f8af5f5a4024455869ea7a896d2364f" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 46121 } } peers { permanent_uuid: "dfa07a7f1fec49beb458bd72df7589e7" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 36845 } } peers { permanent_uuid: "248d7dfbf266486b99f1abfc09afcb8e" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 36851 } }
I20260929 18:40:22.508873 29320 heartbeater.cc:503] Master 127.13.100.62:34215 was elected leader, sending a full tablet report...
I20260929 18:40:22.508883 29607 ts_tablet_manager.cc:1428] T 0b1d9029f71d4111bd46385487da6cad P 0f8af5f5a4024455869ea7a896d2364f: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260929 18:40:22.508931 29606 ts_tablet_manager.cc:1428] T 0b1d9029f71d4111bd46385487da6cad P dfa07a7f1fec49beb458bd72df7589e7: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260929 18:40:22.510052 29588 heartbeater.cc:503] Master 127.13.100.62:34215 was elected leader, sending a full tablet report...
I20260929 18:40:22.529314 29613 raft_consensus.cc:482] T 0b1d9029f71d4111bd46385487da6cad P dfa07a7f1fec49beb458bd72df7589e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:40:22.529537 29613 raft_consensus.cc:504] T 0b1d9029f71d4111bd46385487da6cad P dfa07a7f1fec49beb458bd72df7589e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f8af5f5a4024455869ea7a896d2364f" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 46121 } } peers { permanent_uuid: "dfa07a7f1fec49beb458bd72df7589e7" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 36845 } } peers { permanent_uuid: "248d7dfbf266486b99f1abfc09afcb8e" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 36851 } }
I20260929 18:40:22.530789 29613 leader_election.cc:288] T 0b1d9029f71d4111bd46385487da6cad P dfa07a7f1fec49beb458bd72df7589e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0f8af5f5a4024455869ea7a896d2364f (127.13.100.3:46121), 248d7dfbf266486b99f1abfc09afcb8e (127.13.100.2:36851)
I20260929 18:40:22.534605 29600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:22.531113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.1:51945 (local address 127.13.100.3:46121)
0929 18:40:22.531253 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0929 18:40:22.531257 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:40:22.531366 (+ 109us) server_negotiation.cc:407] Connection header received
0929 18:40:22.531573 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:40:22.531577 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:40:22.531628 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:40:22.531733 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:40:22.532010 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:22.532515 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:22.533566 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:22.534038 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:22.534161 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:40:22.534211 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0929 18:40:22.534399 (+ 188us) server_negotiation.cc:299] Negotiation successful
0929 18:40:22.534476 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 18:40:22.534632 29614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:22.530984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.3:46121 (local address 127.13.100.1:51945)
0929 18:40:22.531273 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0929 18:40:22.531288 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 18:40:22.531436 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:40:22.531748 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:40:22.531751 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:40:22.531771 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:40:22.531899 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:40:22.531904 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:40:22.532628 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:40:22.532632 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:40:22.533440 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:40:22.533448 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:40:22.534152 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:40:22.534156 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:40:22.534256 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:40:22.534279 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 18:40:22.534370 (+ 91us) client_negotiation.cc:240] Negotiation successful
0929 18:40:22.534496 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"mutex_wait_us":23,"thread_start_us":106,"threads_started":1}
I20260929 18:40:22.534849 29604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:22.531524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.1:48895 (local address 127.13.100.2:36851)
0929 18:40:22.531634 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0929 18:40:22.531638 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:40:22.531827 (+ 189us) server_negotiation.cc:407] Connection header received
0929 18:40:22.531944 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:40:22.531946 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:40:22.531988 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:40:22.532079 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:40:22.532323 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:22.532815 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:22.533789 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:22.534252 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:22.534379 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:40:22.534430 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0929 18:40:22.534677 (+ 247us) server_negotiation.cc:299] Negotiation successful
0929 18:40:22.534741 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260929 18:40:22.534868 29615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:22.531384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.2:36851 (local address 127.13.100.1:48895)
0929 18:40:22.531745 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0929 18:40:22.531755 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 18:40:22.531862 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:40:22.532086 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:40:22.532089 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:40:22.532103 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:40:22.532220 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:40:22.532226 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:40:22.532921 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:40:22.532926 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:40:22.533671 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:40:22.533679 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:40:22.534370 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:40:22.534374 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:40:22.534552 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:40:22.534569 (+ 17us) client_negotiation.cc:769] Sending connection context
0929 18:40:22.534659 (+ 90us) client_negotiation.cc:240] Negotiation successful
0929 18:40:22.534727 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"mutex_wait_us":37,"thread_start_us":112,"threads_started":1}
I20260929 18:40:22.535364 29408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b1d9029f71d4111bd46385487da6cad" candidate_uuid: "dfa07a7f1fec49beb458bd72df7589e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "248d7dfbf266486b99f1abfc09afcb8e" is_pre_election: true
I20260929 18:40:22.535338 29542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b1d9029f71d4111bd46385487da6cad" candidate_uuid: "dfa07a7f1fec49beb458bd72df7589e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f8af5f5a4024455869ea7a896d2364f" is_pre_election: true
I20260929 18:40:22.535640 29542 raft_consensus.cc:2454] T 0b1d9029f71d4111bd46385487da6cad P 0f8af5f5a4024455869ea7a896d2364f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dfa07a7f1fec49beb458bd72df7589e7 in term 0.
I20260929 18:40:22.535643 29408 raft_consensus.cc:2454] T 0b1d9029f71d4111bd46385487da6cad P 248d7dfbf266486b99f1abfc09afcb8e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dfa07a7f1fec49beb458bd72df7589e7 in term 0.
I20260929 18:40:22.536113 29207 leader_election.cc:302] T 0b1d9029f71d4111bd46385487da6cad P dfa07a7f1fec49beb458bd72df7589e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 248d7dfbf266486b99f1abfc09afcb8e, dfa07a7f1fec49beb458bd72df7589e7; no voters:
I20260929 18:40:22.536360 29613 raft_consensus.cc:2781] T 0b1d9029f71d4111bd46385487da6cad P dfa07a7f1fec49beb458bd72df7589e7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:40:22.536461 29613 raft_consensus.cc:482] T 0b1d9029f71d4111bd46385487da6cad P dfa07a7f1fec49beb458bd72df7589e7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:40:22.536521 29613 raft_consensus.cc:3037] T 0b1d9029f71d4111bd46385487da6cad P dfa07a7f1fec49beb458bd72df7589e7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:40:22.537539 29613 raft_consensus.cc:504] T 0b1d9029f71d4111bd46385487da6cad P dfa07a7f1fec49beb458bd72df7589e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f8af5f5a4024455869ea7a896d2364f" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 46121 } } peers { permanent_uuid: "dfa07a7f1fec49beb458bd72df7589e7" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 36845 } } peers { permanent_uuid: "248d7dfbf266486b99f1abfc09afcb8e" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 36851 } }
I20260929 18:40:22.537869 29613 leader_election.cc:288] T 0b1d9029f71d4111bd46385487da6cad P dfa07a7f1fec49beb458bd72df7589e7 [CANDIDATE]: Term 1 election: Requested vote from peers 0f8af5f5a4024455869ea7a896d2364f (127.13.100.3:46121), 248d7dfbf266486b99f1abfc09afcb8e (127.13.100.2:36851)
I20260929 18:40:22.538234 29542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b1d9029f71d4111bd46385487da6cad" candidate_uuid: "dfa07a7f1fec49beb458bd72df7589e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f8af5f5a4024455869ea7a896d2364f"
I20260929 18:40:22.538350 29408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b1d9029f71d4111bd46385487da6cad" candidate_uuid: "dfa07a7f1fec49beb458bd72df7589e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "248d7dfbf266486b99f1abfc09afcb8e"
I20260929 18:40:22.538354 29542 raft_consensus.cc:3037] T 0b1d9029f71d4111bd46385487da6cad P 0f8af5f5a4024455869ea7a896d2364f [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:40:22.538497 29408 raft_consensus.cc:3037] T 0b1d9029f71d4111bd46385487da6cad P 248d7dfbf266486b99f1abfc09afcb8e [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:40:22.539407 29542 raft_consensus.cc:2454] T 0b1d9029f71d4111bd46385487da6cad P 0f8af5f5a4024455869ea7a896d2364f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dfa07a7f1fec49beb458bd72df7589e7 in term 1.
I20260929 18:40:22.539698 29207 leader_election.cc:302] T 0b1d9029f71d4111bd46385487da6cad P dfa07a7f1fec49beb458bd72df7589e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f8af5f5a4024455869ea7a896d2364f, dfa07a7f1fec49beb458bd72df7589e7; no voters:
I20260929 18:40:22.539714 29408 raft_consensus.cc:2454] T 0b1d9029f71d4111bd46385487da6cad P 248d7dfbf266486b99f1abfc09afcb8e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dfa07a7f1fec49beb458bd72df7589e7 in term 1.
I20260929 18:40:22.539884 29613 raft_consensus.cc:2781] T 0b1d9029f71d4111bd46385487da6cad P dfa07a7f1fec49beb458bd72df7589e7 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:40:22.540105 29613 raft_consensus.cc:686] T 0b1d9029f71d4111bd46385487da6cad P dfa07a7f1fec49beb458bd72df7589e7 [term 1 LEADER]: Becoming Leader. State: Replica: dfa07a7f1fec49beb458bd72df7589e7, State: Running, Role: LEADER
I20260929 18:40:22.540452 29613 consensus_queue.cc:237] T 0b1d9029f71d4111bd46385487da6cad P dfa07a7f1fec49beb458bd72df7589e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f8af5f5a4024455869ea7a896d2364f" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 46121 } } peers { permanent_uuid: "dfa07a7f1fec49beb458bd72df7589e7" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 36845 } } peers { permanent_uuid: "248d7dfbf266486b99f1abfc09afcb8e" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 36851 } }
I20260929 18:40:22.543740 29126 catalog_manager.cc:5495] T 0b1d9029f71d4111bd46385487da6cad P dfa07a7f1fec49beb458bd72df7589e7 reported cstate change: term changed from 0 to 1, leader changed from <none> to dfa07a7f1fec49beb458bd72df7589e7 (127.13.100.1). New cstate: current_term: 1 leader_uuid: "dfa07a7f1fec49beb458bd72df7589e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f8af5f5a4024455869ea7a896d2364f" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 46121 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dfa07a7f1fec49beb458bd72df7589e7" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 36845 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "248d7dfbf266486b99f1abfc09afcb8e" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 36851 } health_report { overall_health: UNKNOWN } } }
I20260929 18:40:22.566639 29602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:22.563056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51422 (local address 127.13.100.1:36845)
0929 18:40:22.563229 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0929 18:40:22.563234 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:40:22.563246 (+ 12us) server_negotiation.cc:407] Connection header received
0929 18:40:22.563313 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:40:22.563315 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:40:22.563369 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:40:22.563476 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:40:22.563734 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:22.564235 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:22.565141 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:22.565353 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:22.565434 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:40:22.565749 (+ 315us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:40:22.566181 (+ 432us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:40:22.566280 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0929 18:40:22.566389 (+ 109us) server_negotiation.cc:299] Negotiation successful
0929 18:40:22.566474 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 18:40:22.572345 29542 raft_consensus.cc:1261] T 0b1d9029f71d4111bd46385487da6cad P 0f8af5f5a4024455869ea7a896d2364f [term 1 FOLLOWER]: Refusing update from remote peer dfa07a7f1fec49beb458bd72df7589e7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:40:22.572345 29408 raft_consensus.cc:1261] T 0b1d9029f71d4111bd46385487da6cad P 248d7dfbf266486b99f1abfc09afcb8e [term 1 FOLLOWER]: Refusing update from remote peer dfa07a7f1fec49beb458bd72df7589e7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:40:22.573081 29613 consensus_queue.cc:1035] T 0b1d9029f71d4111bd46385487da6cad P dfa07a7f1fec49beb458bd72df7589e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "248d7dfbf266486b99f1abfc09afcb8e" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 36851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:40:22.573263 29616 consensus_queue.cc:1035] T 0b1d9029f71d4111bd46385487da6cad P dfa07a7f1fec49beb458bd72df7589e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f8af5f5a4024455869ea7a896d2364f" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 46121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:40:22.582047 29620 mvcc.cc:205] Tried to move back new op lower bound from 7334736783649046528 to 7334736783528275968. Current Snapshot: MvccSnapshot[applied={T|T < 7334736783649046528}]
I20260929 18:40:22.583161 29619 mvcc.cc:205] Tried to move back new op lower bound from 7334736783649046528 to 7334736783528275968. Current Snapshot: MvccSnapshot[applied={T|T < 7334736783649046528}]
I20260929 18:40:22.583877 29621 mvcc.cc:205] Tried to move back new op lower bound from 7334736783649046528 to 7334736783528275968. Current Snapshot: MvccSnapshot[applied={T|T < 7334736783649046528}]
I20260929 18:40:22.590216 29129 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52076:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:40:22.590400 29129 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52076:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:40:22.593003 29129 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 55348a079a5f4d1795e48eda404bb105: Sending DeleteTablet for 3 replicas of tablet 0b1d9029f71d4111bd46385487da6cad
I20260929 18:40:22.593807 29254 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b1d9029f71d4111bd46385487da6cad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:40:22 UTC) from {username='kudu', principal='kudu/127.13.100.62@KRBTEST.COM'} at 127.0.0.1:51414
I20260929 18:40:22.593827 29388 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b1d9029f71d4111bd46385487da6cad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:40:22 UTC) from {username='kudu', principal='kudu/127.13.100.62@KRBTEST.COM'} at 127.0.0.1:58122
I20260929 18:40:22.593796 29522 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b1d9029f71d4111bd46385487da6cad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:40:22 UTC) from {username='kudu', principal='kudu/127.13.100.62@KRBTEST.COM'} at 127.0.0.1:34970
I20260929 18:40:22.594409 29629 tablet_replica.cc:330] T 0b1d9029f71d4111bd46385487da6cad P dfa07a7f1fec49beb458bd72df7589e7: stopping tablet replica
I20260929 18:40:22.594499 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 29191
I20260929 18:40:22.594831 29629 raft_consensus.cc:2229] T 0b1d9029f71d4111bd46385487da6cad P dfa07a7f1fec49beb458bd72df7589e7 [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:40:22.595209 29631 tablet_replica.cc:330] T 0b1d9029f71d4111bd46385487da6cad P 0f8af5f5a4024455869ea7a896d2364f: stopping tablet replica
I20260929 18:40:22.595203 29630 tablet_replica.cc:330] T 0b1d9029f71d4111bd46385487da6cad P 248d7dfbf266486b99f1abfc09afcb8e: stopping tablet replica
I20260929 18:40:22.595485 29630 raft_consensus.cc:2229] T 0b1d9029f71d4111bd46385487da6cad P 248d7dfbf266486b99f1abfc09afcb8e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:40:22.595507 29631 raft_consensus.cc:2229] T 0b1d9029f71d4111bd46385487da6cad P 0f8af5f5a4024455869ea7a896d2364f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:40:22.595803 29630 raft_consensus.cc:2258] T 0b1d9029f71d4111bd46385487da6cad P 248d7dfbf266486b99f1abfc09afcb8e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:40:22.595803 29631 raft_consensus.cc:2258] T 0b1d9029f71d4111bd46385487da6cad P 0f8af5f5a4024455869ea7a896d2364f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:40:22.597623 29631 ts_tablet_manager.cc:1905] T 0b1d9029f71d4111bd46385487da6cad P 0f8af5f5a4024455869ea7a896d2364f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:40:22.598743 29630 ts_tablet_manager.cc:1905] T 0b1d9029f71d4111bd46385487da6cad P 248d7dfbf266486b99f1abfc09afcb8e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:40:22.600935 29630 ts_tablet_manager.cc:1918] T 0b1d9029f71d4111bd46385487da6cad P 248d7dfbf266486b99f1abfc09afcb8e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:40:22.601064 29630 log.cc:1196] T 0b1d9029f71d4111bd46385487da6cad P 248d7dfbf266486b99f1abfc09afcb8e: Deleting WAL directory at /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/ts-1/wal/wals/0b1d9029f71d4111bd46385487da6cad
I20260929 18:40:22.601161 29631 ts_tablet_manager.cc:1918] T 0b1d9029f71d4111bd46385487da6cad P 0f8af5f5a4024455869ea7a896d2364f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:40:22.601263 29631 log.cc:1196] T 0b1d9029f71d4111bd46385487da6cad P 0f8af5f5a4024455869ea7a896d2364f: Deleting WAL directory at /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/ts-2/wal/wals/0b1d9029f71d4111bd46385487da6cad
I20260929 18:40:22.601457 29630 ts_tablet_manager.cc:1939] T 0b1d9029f71d4111bd46385487da6cad P 248d7dfbf266486b99f1abfc09afcb8e: Deleting consensus metadata
I20260929 18:40:22.601665 29631 ts_tablet_manager.cc:1939] T 0b1d9029f71d4111bd46385487da6cad P 0f8af5f5a4024455869ea7a896d2364f: Deleting consensus metadata
W20260929 18:40:22.602608 29114 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.100.1:36845 (error 108)
I20260929 18:40:22.603022 29113 catalog_manager.cc:4841] TS 0f8af5f5a4024455869ea7a896d2364f (127.13.100.3:46121): tablet 0b1d9029f71d4111bd46385487da6cad (table test-table [id=6bf868ed94424da0a4856b77104f7a47]) successfully deleted
I20260929 18:40:22.603240 29113 catalog_manager.cc:4841] TS 248d7dfbf266486b99f1abfc09afcb8e (127.13.100.2:36851): tablet 0b1d9029f71d4111bd46385487da6cad (table test-table [id=6bf868ed94424da0a4856b77104f7a47]) successfully deleted
I20260929 18:40:22.603356 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 29325
I20260929 18:40:22.603762 29603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:22.603410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.1:36845 (local address 127.0.0.1:51438)
0929 18:40:22.603596 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0929 18:40:22.603667 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.1:36845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
W20260929 18:40:22.604146 29114 catalog_manager.cc:4584] TS dfa07a7f1fec49beb458bd72df7589e7 (127.13.100.1:36845): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0b1d9029f71d4111bd46385487da6cad: Network error: Client connection negotiation failed: client connection to 127.13.100.1:36845: connect: Connection refused (error 111)
I20260929 18:40:22.609509 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 29459
I20260929 18:40:22.615717 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 29097
I20260929 18:40:42.622334 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.100.62:34215
--webserver_interface=127.13.100.62
--webserver_port=42543
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.62.keytab
--principal=kudu/127.13.100.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.100.20:44697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.100.62:34215
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:40:42.730366 29633 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:40:42.730659 29633 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:40:42.730727 29633 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:40:42.734676 29633 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 18:40:42.734741 29633 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 18:40:42.734772 29633 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 18:40:42.734823 29633 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:40:42.734869 29633 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:40:42.734910 29633 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:40:42.734946 29633 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:40:42.734977 29633 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:40:42.740445 29633 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.100.20:44697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.13.100.62:34215
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.62.keytab
--principal=kudu/127.13.100.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.100.62:34215
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.100.62
--webserver_port=42543
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29633
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:40:42.741469 29633 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:40:42.742482 29633 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:40:42.755623 29639 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:40:42.755820 29640 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:40:42.755719 29642 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:40:42.755672 29633 server_base.cc:1034] running on GCE node
I20260929 18:40:42.756532 29633 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:40:42.757682 29633 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:40:42.758875 29633 hybrid_clock.cc:648] HybridClock initialized: now 1790707242758852 us; error 79 us; skew 500 ppm
Sep 29 18:40:42 dist-test-slave-j1xs krb5kdc[29084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707242, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:40:42.761683 29633 init.cc:377] Logged in from keytab as kudu/127.13.100.62@KRBTEST.COM (short username kudu)
I20260929 18:40:42.762637 29633 webserver.cc:460] Webserver started at http://127.13.100.62:42543/ using document root <none> and password file <none>
I20260929 18:40:42.763218 29633 fs_manager.cc:359] Metadata directory not provided
I20260929 18:40:42.763326 29633 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:40:42.767192 29633 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.002s
I20260929 18:40:42.769306 29649 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:40:42.770524 29633 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260929 18:40:42.770663 29633 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/master-0/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/master-0/wal
uuid: "55348a079a5f4d1795e48eda404bb105"
format_stamp: "Formatted at 2026-09-29 18:40:21 on dist-test-slave-j1xs"
server_key: "3028baaa3654086d63191a66f22284da"
server_key_iv: "f057f747b98d8b08403bbef039e93785"
server_key_version: "encryptionkey@0"
I20260929 18:40:42.771257 29633 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:40:42.794320 29633 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:40:42.797770 29633 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:40:42.798061 29633 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:40:42.805570 29633 rpc_server.cc:307] RPC server started. Bound to: 127.13.100.62:34215
I20260929 18:40:42.805853 29701 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.100.62:34215 every 8 connection(s)
I20260929 18:40:42.806828 29633 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
I20260929 18:40:42.808848 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 29633
I20260929 18:40:42.809397 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.100.1:36845
--local_ip_for_outbound_sockets=127.13.100.1
--tserver_master_addrs=127.13.100.62:34215
--webserver_port=38303
--webserver_interface=127.13.100.1
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.1.keytab
--principal=kudu/127.13.100.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.100.20:44697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 18:40:42.812207 29702 sys_catalog.cc:263] Verifying existing consensus state
I20260929 18:40:42.814976 29702 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55348a079a5f4d1795e48eda404bb105: Bootstrap starting.
I20260929 18:40:42.838356 29702 log.cc:824] T 00000000000000000000000000000000 P 55348a079a5f4d1795e48eda404bb105: Log is configured to *not* fsync() on all Append() calls
I20260929 18:40:42.847302 29702 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55348a079a5f4d1795e48eda404bb105: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 18:40:42.847707 29702 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55348a079a5f4d1795e48eda404bb105: Bootstrap complete.
I20260929 18:40:42.851759 29702 raft_consensus.cc:348] T 00000000000000000000000000000000 P 55348a079a5f4d1795e48eda404bb105 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55348a079a5f4d1795e48eda404bb105" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 34215 } }
I20260929 18:40:42.852173 29702 raft_consensus.cc:729] T 00000000000000000000000000000000 P 55348a079a5f4d1795e48eda404bb105 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55348a079a5f4d1795e48eda404bb105, State: Initialized, Role: FOLLOWER
I20260929 18:40:42.852649 29702 consensus_queue.cc:260] T 00000000000000000000000000000000 P 55348a079a5f4d1795e48eda404bb105 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55348a079a5f4d1795e48eda404bb105" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 34215 } }
I20260929 18:40:42.852799 29702 raft_consensus.cc:388] T 00000000000000000000000000000000 P 55348a079a5f4d1795e48eda404bb105 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:40:42.852885 29702 raft_consensus.cc:482] T 00000000000000000000000000000000 P 55348a079a5f4d1795e48eda404bb105 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:40:42.853008 29702 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 55348a079a5f4d1795e48eda404bb105 [term 1 FOLLOWER]: Advancing to term 2
I20260929 18:40:42.855017 29702 raft_consensus.cc:504] T 00000000000000000000000000000000 P 55348a079a5f4d1795e48eda404bb105 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55348a079a5f4d1795e48eda404bb105" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 34215 } }
I20260929 18:40:42.855355 29702 leader_election.cc:302] T 00000000000000000000000000000000 P 55348a079a5f4d1795e48eda404bb105 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 55348a079a5f4d1795e48eda404bb105; no voters:
I20260929 18:40:42.855724 29702 leader_election.cc:288] T 00000000000000000000000000000000 P 55348a079a5f4d1795e48eda404bb105 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 18:40:42.855806 29706 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 55348a079a5f4d1795e48eda404bb105 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 18:40:42.856014 29706 raft_consensus.cc:686] T 00000000000000000000000000000000 P 55348a079a5f4d1795e48eda404bb105 [term 2 LEADER]: Becoming Leader. State: Replica: 55348a079a5f4d1795e48eda404bb105, State: Running, Role: LEADER
I20260929 18:40:42.856369 29706 consensus_queue.cc:237] T 00000000000000000000000000000000 P 55348a079a5f4d1795e48eda404bb105 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55348a079a5f4d1795e48eda404bb105" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 34215 } }
I20260929 18:40:42.856959 29702 sys_catalog.cc:564] T 00000000000000000000000000000000 P 55348a079a5f4d1795e48eda404bb105 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:40:42.858023 29708 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55348a079a5f4d1795e48eda404bb105 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 55348a079a5f4d1795e48eda404bb105. Latest consensus state: current_term: 2 leader_uuid: "55348a079a5f4d1795e48eda404bb105" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55348a079a5f4d1795e48eda404bb105" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 34215 } } }
I20260929 18:40:42.858167 29708 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55348a079a5f4d1795e48eda404bb105 [sys.catalog]: This master's current role is: LEADER
I20260929 18:40:42.858417 29707 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55348a079a5f4d1795e48eda404bb105 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "55348a079a5f4d1795e48eda404bb105" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55348a079a5f4d1795e48eda404bb105" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 34215 } } }
I20260929 18:40:42.858487 29707 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55348a079a5f4d1795e48eda404bb105 [sys.catalog]: This master's current role is: LEADER
I20260929 18:40:42.858502 29715 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:40:42.863169 29715 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:40:42.863857 29715 catalog_manager.cc:1259] Loaded cluster ID: e62d58bc40c54a52be38c7f6f3340271
I20260929 18:40:42.863919 29715 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:40:42.866866 29715 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:40:42.867599 29715 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 55348a079a5f4d1795e48eda404bb105: Loaded TSK: 0
I20260929 18:40:42.882169 29715 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 55348a079a5f4d1795e48eda404bb105: Generated new TSK 1
I20260929 18:40:42.882532 29715 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 18:40:42.935737 29704 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:40:42.936048 29704 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:40:42.936131 29704 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:40:42.940510 29704 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:40:42.940626 29704 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:40:42.940725 29704 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.100.1
I20260929 18:40:42.946399 29704 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.100.20:44697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.1.keytab
--principal=kudu/127.13.100.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.100.1:36845
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.100.1
--webserver_port=38303
--tserver_master_addrs=127.13.100.62:34215
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29704
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.100.1
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:40:42.947456 29704 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:40:42.948463 29704 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:40:42.958843 29730 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:40:42.958830 29729 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:40:42.958824 29732 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:40:42.959313 29704 server_base.cc:1034] running on GCE node
I20260929 18:40:42.959796 29704 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:40:42.960534 29704 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:40:42.961735 29704 hybrid_clock.cc:648] HybridClock initialized: now 1790707242961706 us; error 48 us; skew 500 ppm
Sep 29 18:40:42 dist-test-slave-j1xs krb5kdc[29084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707242, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:40:42.964958 29704 init.cc:377] Logged in from keytab as kudu/127.13.100.1@KRBTEST.COM (short username kudu)
I20260929 18:40:42.965785 29704 webserver.cc:460] Webserver started at http://127.13.100.1:38303/ using document root <none> and password file <none>
I20260929 18:40:42.966426 29704 fs_manager.cc:359] Metadata directory not provided
I20260929 18:40:42.966506 29704 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:40:42.970314 29704 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 18:40:42.972805 29739 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:40:42.974321 29704 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:40:42.974479 29704 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/ts-0/wal
uuid: "dfa07a7f1fec49beb458bd72df7589e7"
format_stamp: "Formatted at 2026-09-29 18:40:21 on dist-test-slave-j1xs"
server_key: "678290f32b6284b80e3321b76a46581c"
server_key_iv: "546083d1f5bf912f0549e28f3e4e8826"
server_key_version: "encryptionkey@0"
I20260929 18:40:42.975060 29704 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:40:42.997819 29704 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:40:43.001148 29704 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:40:43.001363 29704 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:40:43.002022 29704 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:40:43.003157 29746 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260929 18:40:43.008390 29704 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 18:40:43.008495 29704 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260929 18:40:43.008606 29704 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 18:40:43.010439 29704 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 18:40:43.010485 29704 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.002s sys 0.000s
I20260929 18:40:43.010807 29746 tablet_bootstrap.cc:492] T 0b1d9029f71d4111bd46385487da6cad P dfa07a7f1fec49beb458bd72df7589e7: Bootstrap starting.
I20260929 18:40:43.020633 29704 rpc_server.cc:307] RPC server started. Bound to: 127.13.100.1:36845
I20260929 18:40:43.020710 29853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.100.1:36845 every 8 connection(s)
I20260929 18:40:43.021591 29704 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/ts-0/data/info.pb
I20260929 18:40:43.027315 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 29704
I20260929 18:40:43.027951 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.100.2:36851
--local_ip_for_outbound_sockets=127.13.100.2
--tserver_master_addrs=127.13.100.62:34215
--webserver_port=36695
--webserver_interface=127.13.100.2
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.2.keytab
--principal=kudu/127.13.100.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.100.20:44697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 18:40:43.030340 29746 log.cc:824] T 0b1d9029f71d4111bd46385487da6cad P dfa07a7f1fec49beb458bd72df7589e7: Log is configured to *not* fsync() on all Append() calls
Sep 29 18:40:43 dist-test-slave-j1xs krb5kdc[29084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707242, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.1@KRBTEST.COM for kudu/127.13.100.62@KRBTEST.COM
I20260929 18:40:43.038651 29746 tablet_bootstrap.cc:492] T 0b1d9029f71d4111bd46385487da6cad P dfa07a7f1fec49beb458bd72df7589e7: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 18:40:43.039088 29746 tablet_bootstrap.cc:492] T 0b1d9029f71d4111bd46385487da6cad P dfa07a7f1fec49beb458bd72df7589e7: Bootstrap complete.
I20260929 18:40:43.040427 29746 ts_tablet_manager.cc:1397] T 0b1d9029f71d4111bd46385487da6cad P dfa07a7f1fec49beb458bd72df7589e7: Time spent bootstrapping tablet: real 0.030s user 0.023s sys 0.004s
I20260929 18:40:43.040503 29857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:43.023541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:34215 (local address 127.13.100.1:42317)
0929 18:40:43.024172 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0929 18:40:43.024223 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 18:40:43.025020 (+ 797us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:40:43.026322 (+ 1302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:40:43.026329 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:40:43.026683 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:40:43.027138 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:40:43.027152 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:40:43.028652 (+ 1500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:40:43.028655 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:40:43.029720 (+ 1065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:40:43.029734 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:40:43.030738 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:40:43.030745 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:40:43.030974 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:40:43.031787 (+ 813us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:40:43.031812 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:40:43.034033 (+ 2221us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:40:43.036890 (+ 2857us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:40:43.036896 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:40:43.036907 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:40:43.037238 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:40:43.037671 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:40:43.037674 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:40:43.037676 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:40:43.037834 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:40:43.038652 (+ 818us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:40:43.038657 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:40:43.038966 (+ 309us) client_negotiation.cc:769] Sending connection context
0929 18:40:43.039175 (+ 209us) client_negotiation.cc:240] Negotiation successful
0929 18:40:43.039515 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":145,"threads_started":1}
I20260929 18:40:43.040648 29856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:43.023660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.1:42317 (local address 127.13.100.62:34215)
0929 18:40:43.024056 (+ 396us) server_negotiation.cc:206] Beginning negotiation
0929 18:40:43.024078 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0929 18:40:43.024363 (+ 285us) server_negotiation.cc:407] Connection header received
0929 18:40:43.025467 (+ 1104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:40:43.025482 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:40:43.025867 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:40:43.026284 (+ 417us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:40:43.027311 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:43.028474 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:43.029865 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:43.030587 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:43.030779 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:40:43.034269 (+ 3490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:40:43.034278 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:40:43.034299 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:40:43.034339 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:40:43.036751 (+ 2412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:40:43.037370 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:40:43.037378 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:40:43.037385 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:40:43.037556 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:40:43.037986 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:40:43.037991 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:40:43.037995 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:40:43.038514 (+ 519us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:40:43.038669 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0929 18:40:43.039371 (+ 702us) server_negotiation.cc:299] Negotiation successful
0929 18:40:43.039693 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":112,"threads_started":1}
I20260929 18:40:43.043552 29854 heartbeater.cc:347] Connected to a master server at 127.13.100.62:34215
I20260929 18:40:43.043282 29746 raft_consensus.cc:348] T 0b1d9029f71d4111bd46385487da6cad P dfa07a7f1fec49beb458bd72df7589e7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f8af5f5a4024455869ea7a896d2364f" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 46121 } } peers { permanent_uuid: "dfa07a7f1fec49beb458bd72df7589e7" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 36845 } } peers { permanent_uuid: "248d7dfbf266486b99f1abfc09afcb8e" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 36851 } }
I20260929 18:40:43.043792 29854 heartbeater.cc:464] Registering TS with master...
I20260929 18:40:43.043864 29746 raft_consensus.cc:729] T 0b1d9029f71d4111bd46385487da6cad P dfa07a7f1fec49beb458bd72df7589e7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfa07a7f1fec49beb458bd72df7589e7, State: Initialized, Role: FOLLOWER
I20260929 18:40:43.044337 29854 heartbeater.cc:511] Master 127.13.100.62:34215 requested a full tablet report, sending...
I20260929 18:40:43.044351 29746 consensus_queue.cc:260] T 0b1d9029f71d4111bd46385487da6cad P dfa07a7f1fec49beb458bd72df7589e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f8af5f5a4024455869ea7a896d2364f" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 46121 } } peers { permanent_uuid: "dfa07a7f1fec49beb458bd72df7589e7" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 36845 } } peers { permanent_uuid: "248d7dfbf266486b99f1abfc09afcb8e" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 36851 } }
I20260929 18:40:43.045362 29746 ts_tablet_manager.cc:1428] T 0b1d9029f71d4111bd46385487da6cad P dfa07a7f1fec49beb458bd72df7589e7: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260929 18:40:43.046819 29666 ts_manager.cc:194] Registered new tserver with Master: dfa07a7f1fec49beb458bd72df7589e7 (127.13.100.1:36845)
I20260929 18:40:43.050381 29666 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.100.1@KRBTEST.COM'} at 127.13.100.1:42317
I20260929 18:40:43.052788 29854 heartbeater.cc:503] Master 127.13.100.62:34215 was elected leader, sending a full tablet report...
Sep 29 18:40:43 dist-test-slave-j1xs krb5kdc[29084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707242, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.62@KRBTEST.COM for kudu/127.13.100.1@KRBTEST.COM
I20260929 18:40:43.060815 29861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:43.049428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.1:36845 (local address 127.0.0.1:40478)
0929 18:40:43.050355 (+ 927us) negotiation.cc:107] Waiting for socket to connect
0929 18:40:43.050399 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 18:40:43.050535 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:40:43.051044 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:40:43.051051 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:40:43.051585 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:40:43.051717 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:40:43.051724 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:40:43.053010 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:40:43.053013 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:40:43.053740 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:40:43.053749 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:40:43.054476 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:40:43.054480 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:40:43.054585 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:40:43.055127 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:40:43.055142 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:40:43.056749 (+ 1607us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:40:43.059114 (+ 2365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:40:43.059122 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:40:43.059126 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:40:43.059390 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:40:43.059702 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:40:43.059704 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:40:43.059706 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:40:43.059782 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:40:43.060294 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:40:43.060299 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:40:43.060430 (+ 131us) client_negotiation.cc:769] Sending connection context
0929 18:40:43.060544 (+ 114us) client_negotiation.cc:240] Negotiation successful
0929 18:40:43.060688 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":744,"thread_start_us":125,"threads_started":1}
I20260929 18:40:43.060971 29862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:43.049507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40478 (local address 127.13.100.1:36845)
0929 18:40:43.050211 (+ 704us) server_negotiation.cc:206] Beginning negotiation
0929 18:40:43.050221 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 18:40:43.050561 (+ 340us) server_negotiation.cc:407] Connection header received
0929 18:40:43.050685 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:40:43.050690 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:40:43.050883 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:40:43.051022 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:40:43.051849 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:43.052859 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:43.053860 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:43.054264 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:43.054371 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:40:43.056892 (+ 2521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:40:43.056901 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:40:43.056913 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:40:43.056939 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:40:43.059001 (+ 2062us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:40:43.059503 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:40:43.059509 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:40:43.059511 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:40:43.059608 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:40:43.059878 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:40:43.059881 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:40:43.059883 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:40:43.060174 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:40:43.060300 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0929 18:40:43.060647 (+ 347us) server_negotiation.cc:299] Negotiation successful
0929 18:40:43.060804 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":611,"thread_start_us":76,"threads_started":1}
I20260929 18:40:43.062016 29788 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b1d9029f71d4111bd46385487da6cad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:40:22 UTC) from {username='kudu', principal='kudu/127.13.100.62@KRBTEST.COM'} at 127.0.0.1:40478
I20260929 18:40:43.062938 29863 tablet_replica.cc:330] stopping tablet replica
I20260929 18:40:43.063248 29863 raft_consensus.cc:2229] T 0b1d9029f71d4111bd46385487da6cad P dfa07a7f1fec49beb458bd72df7589e7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:40:43.063496 29863 raft_consensus.cc:2258] T 0b1d9029f71d4111bd46385487da6cad P dfa07a7f1fec49beb458bd72df7589e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:40:43.064556 29863 ts_tablet_manager.cc:1905] T 0b1d9029f71d4111bd46385487da6cad P dfa07a7f1fec49beb458bd72df7589e7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:40:43.069319 29863 ts_tablet_manager.cc:1918] T 0b1d9029f71d4111bd46385487da6cad P dfa07a7f1fec49beb458bd72df7589e7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:40:43.069446 29863 log.cc:1196] T 0b1d9029f71d4111bd46385487da6cad P dfa07a7f1fec49beb458bd72df7589e7: Deleting WAL directory at /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/ts-0/wal/wals/0b1d9029f71d4111bd46385487da6cad
I20260929 18:40:43.069758 29863 ts_tablet_manager.cc:1939] T 0b1d9029f71d4111bd46385487da6cad P dfa07a7f1fec49beb458bd72df7589e7: Deleting consensus metadata
I20260929 18:40:43.071730 29651 catalog_manager.cc:4841] TS dfa07a7f1fec49beb458bd72df7589e7 (127.13.100.1:36845): tablet 0b1d9029f71d4111bd46385487da6cad (table test-table [id=6bf868ed94424da0a4856b77104f7a47]) successfully deleted
W20260929 18:40:43.154073 29858 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:40:43.154441 29858 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:40:43.154503 29858 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:40:43.159344 29858 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:40:43.159411 29858 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:40:43.159476 29858 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.100.2
I20260929 18:40:43.164989 29858 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.100.20:44697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.2.keytab
--principal=kudu/127.13.100.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.100.2:36851
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.100.2
--webserver_port=36695
--tserver_master_addrs=127.13.100.62:34215
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29858
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.100.2
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:40:43.166589 29858 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:40:43.167613 29858 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:40:43.180469 29871 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:40:43.180545 29868 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:40:43.180557 29869 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:40:43.180975 29858 server_base.cc:1034] running on GCE node
I20260929 18:40:43.181448 29858 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:40:43.182297 29858 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:40:43.183490 29858 hybrid_clock.cc:648] HybridClock initialized: now 1790707243183453 us; error 57 us; skew 500 ppm
Sep 29 18:40:43 dist-test-slave-j1xs krb5kdc[29084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707243, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:40:43.186616 29858 init.cc:377] Logged in from keytab as kudu/127.13.100.2@KRBTEST.COM (short username kudu)
I20260929 18:40:43.187534 29858 webserver.cc:460] Webserver started at http://127.13.100.2:36695/ using document root <none> and password file <none>
I20260929 18:40:43.188133 29858 fs_manager.cc:359] Metadata directory not provided
I20260929 18:40:43.188239 29858 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:40:43.192107 29858 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 18:40:43.194265 29878 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:40:43.195595 29858 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260929 18:40:43.195760 29858 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/ts-1/wal
uuid: "248d7dfbf266486b99f1abfc09afcb8e"
format_stamp: "Formatted at 2026-09-29 18:40:22 on dist-test-slave-j1xs"
server_key: "dcf8d37d7b3add7c0354687c0d3bfd2e"
server_key_iv: "56e823f10c16a4a9bb098bf07a198f47"
server_key_version: "encryptionkey@0"
I20260929 18:40:43.196306 29858 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:40:43.213821 29858 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:40:43.217198 29858 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:40:43.217411 29858 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:40:43.218122 29858 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:40:43.218955 29858 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:40:43.219003 29858 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:40:43.219043 29858 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:40:43.219099 29858 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:40:43.227645 29858 rpc_server.cc:307] RPC server started. Bound to: 127.13.100.2:36851
I20260929 18:40:43.227694 29991 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.100.2:36851 every 8 connection(s)
I20260929 18:40:43.228953 29858 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/ts-1/data/info.pb
I20260929 18:40:43.235625 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 29858
I20260929 18:40:43.236142 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.100.3:46121
--local_ip_for_outbound_sockets=127.13.100.3
--tserver_master_addrs=127.13.100.62:34215
--webserver_port=34335
--webserver_interface=127.13.100.3
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.3.keytab
--principal=kudu/127.13.100.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.100.20:44697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 18:40:43 dist-test-slave-j1xs krb5kdc[29084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707243, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.2@KRBTEST.COM for kudu/127.13.100.62@KRBTEST.COM
I20260929 18:40:43.243788 29856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:43.231654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.2:55345 (local address 127.13.100.62:34215)
0929 18:40:43.231896 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0929 18:40:43.231899 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:40:43.232685 (+ 786us) server_negotiation.cc:407] Connection header received
0929 18:40:43.233961 (+ 1276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:40:43.233964 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:40:43.234018 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:40:43.234100 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:40:43.235290 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:43.235901 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:43.237156 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:43.237557 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:43.237676 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:40:43.240223 (+ 2547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:40:43.240232 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:40:43.240234 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:40:43.240257 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:40:43.241726 (+ 1469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:40:43.242265 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:40:43.242270 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:40:43.242272 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:40:43.242353 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:40:43.242709 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:40:43.242712 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:40:43.242713 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:40:43.242891 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:40:43.242994 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 18:40:43.243483 (+ 489us) server_negotiation.cc:299] Negotiation successful
0929 18:40:43.243613 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125}
I20260929 18:40:43.244669 29994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:43.232056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:34215 (local address 127.13.100.2:55345)
0929 18:40:43.232508 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0929 18:40:43.232552 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 18:40:43.233739 (+ 1187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:40:43.234263 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:40:43.234269 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:40:43.234609 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:40:43.235093 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:40:43.235106 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:40:43.236056 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:40:43.236059 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:40:43.237012 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:40:43.237021 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:40:43.237715 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:40:43.237718 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:40:43.237862 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:40:43.238586 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:40:43.238602 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:40:43.240057 (+ 1455us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:40:43.241846 (+ 1789us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:40:43.241852 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:40:43.241866 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:40:43.242160 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:40:43.242446 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:40:43.242451 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:40:43.242453 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:40:43.242604 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:40:43.243004 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:40:43.243013 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:40:43.243247 (+ 234us) client_negotiation.cc:769] Sending connection context
0929 18:40:43.243428 (+ 181us) client_negotiation.cc:240] Negotiation successful
0929 18:40:43.243759 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":107,"threads_started":1}
I20260929 18:40:43.245868 29992 heartbeater.cc:347] Connected to a master server at 127.13.100.62:34215
I20260929 18:40:43.246162 29992 heartbeater.cc:464] Registering TS with master...
I20260929 18:40:43.246651 29992 heartbeater.cc:511] Master 127.13.100.62:34215 requested a full tablet report, sending...
I20260929 18:40:43.247722 29666 ts_manager.cc:194] Registered new tserver with Master: 248d7dfbf266486b99f1abfc09afcb8e (127.13.100.2:36851)
I20260929 18:40:43.248471 29666 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.100.2@KRBTEST.COM'} at 127.13.100.2:55345
W20260929 18:40:43.347816 29995 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:40:43.348059 29995 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:40:43.348101 29995 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:40:43.352034 29995 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:40:43.352084 29995 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:40:43.352140 29995 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.100.3
I20260929 18:40:43.357543 29995 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.100.20:44697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.3.keytab
--principal=kudu/127.13.100.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.100.3:46121
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.100.3
--webserver_port=34335
--tserver_master_addrs=127.13.100.62:34215
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29995
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.100.3
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:40:43.358692 29995 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:40:43.359724 29995 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:40:43.369309 30003 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:40:43.369313 30001 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:40:43.369349 30000 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:40:43.369498 29995 server_base.cc:1034] running on GCE node
I20260929 18:40:43.370021 29995 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:40:43.370669 29995 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:40:43.371850 29995 hybrid_clock.cc:648] HybridClock initialized: now 1790707243371815 us; error 52 us; skew 500 ppm
Sep 29 18:40:43 dist-test-slave-j1xs krb5kdc[29084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707243, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:40:43.375301 29995 init.cc:377] Logged in from keytab as kudu/127.13.100.3@KRBTEST.COM (short username kudu)
I20260929 18:40:43.376204 29995 webserver.cc:460] Webserver started at http://127.13.100.3:34335/ using document root <none> and password file <none>
I20260929 18:40:43.376767 29995 fs_manager.cc:359] Metadata directory not provided
I20260929 18:40:43.376873 29995 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:40:43.380847 29995 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 18:40:43.382951 30010 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:40:43.384171 29995 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:40:43.384302 29995 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/ts-2/wal
uuid: "0f8af5f5a4024455869ea7a896d2364f"
format_stamp: "Formatted at 2026-09-29 18:40:22 on dist-test-slave-j1xs"
server_key: "522c02267cf80687cb19146bab7805ee"
server_key_iv: "b781fd704777451fefe12e45a4893748"
server_key_version: "encryptionkey@0"
I20260929 18:40:43.384829 29995 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:40:43.414482 29995 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:40:43.417873 29995 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:40:43.418143 29995 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:40:43.418752 29995 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:40:43.419565 29995 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:40:43.419636 29995 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:40:43.419708 29995 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:40:43.419750 29995 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:40:43.428598 29995 rpc_server.cc:307] RPC server started. Bound to: 127.13.100.3:46121
I20260929 18:40:43.428639 30123 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.100.3:46121 every 8 connection(s)
I20260929 18:40:43.429759 29995 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/ts-2/data/info.pb
I20260929 18:40:43.433734 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 29995
Sep 29 18:40:43 dist-test-slave-j1xs krb5kdc[29084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707243, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.3@KRBTEST.COM for kudu/127.13.100.62@KRBTEST.COM
I20260929 18:40:43.442240 30128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:43.434453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44768 (local address 127.13.100.62:34215)
0929 18:40:43.434701 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0929 18:40:43.434704 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:40:43.434791 (+ 87us) server_negotiation.cc:407] Connection header received
0929 18:40:43.434920 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:40:43.434923 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:40:43.434958 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:40:43.435032 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:40:43.435772 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:43.436289 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:43.437120 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:43.437346 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:43.437452 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:40:43.438909 (+ 1457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:40:43.438913 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:40:43.438916 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:40:43.438942 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:40:43.440553 (+ 1611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:40:43.440993 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:40:43.440998 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:40:43.441000 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:40:43.441110 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:40:43.441399 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:40:43.441403 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:40:43.441404 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:40:43.441612 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:40:43.441836 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0929 18:40:43.441930 (+ 94us) server_negotiation.cc:299] Negotiation successful
0929 18:40:43.442051 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":84,"threads_started":1}
I20260929 18:40:43.444000 29856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:43.431637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.3:34885 (local address 127.13.100.62:34215)
0929 18:40:43.431769 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 18:40:43.431772 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:40:43.432715 (+ 943us) server_negotiation.cc:407] Connection header received
0929 18:40:43.433574 (+ 859us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:40:43.433577 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:40:43.433623 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:40:43.433698 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:40:43.435197 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:43.435696 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:43.436978 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:43.437208 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:43.437317 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:40:43.440312 (+ 2995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:40:43.440315 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:40:43.440317 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:40:43.440346 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:40:43.441951 (+ 1605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:40:43.442559 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:40:43.442561 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:40:43.442563 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:40:43.442663 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:40:43.443032 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:40:43.443035 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:40:43.443037 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:40:43.443222 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:40:43.443314 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0929 18:40:43.443781 (+ 467us) server_negotiation.cc:299] Negotiation successful
0929 18:40:43.443887 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 18:40:43.444893 30126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:43.431927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:34215 (local address 127.13.100.3:34885)
0929 18:40:43.432556 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0929 18:40:43.432594 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 18:40:43.433325 (+ 731us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:40:43.433882 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:40:43.433916 (+ 34us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:40:43.434402 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:40:43.435057 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:40:43.435075 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:40:43.435842 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:40:43.435844 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:40:43.436845 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:40:43.436855 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:40:43.437352 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:40:43.437365 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:40:43.437522 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:40:43.438277 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:40:43.438292 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:40:43.440149 (+ 1857us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:40:43.442070 (+ 1921us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:40:43.442076 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:40:43.442090 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:40:43.442448 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:40:43.442764 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:40:43.442767 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:40:43.442769 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:40:43.442933 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:40:43.443320 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:40:43.443325 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:40:43.443558 (+ 233us) client_negotiation.cc:769] Sending connection context
0929 18:40:43.443743 (+ 185us) client_negotiation.cc:240] Negotiation successful
0929 18:40:43.444082 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":108,"threads_started":1}
I20260929 18:40:43.445915 30124 heartbeater.cc:347] Connected to a master server at 127.13.100.62:34215
I20260929 18:40:43.446110 30124 heartbeater.cc:464] Registering TS with master...
I20260929 18:40:43.446497 30124 heartbeater.cc:511] Master 127.13.100.62:34215 requested a full tablet report, sending...
I20260929 18:40:43.447341 29666 ts_manager.cc:194] Registered new tserver with Master: 0f8af5f5a4024455869ea7a896d2364f (127.13.100.3:46121)
I20260929 18:40:43.447809 29666 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.100.3@KRBTEST.COM'} at 127.13.100.3:34885
I20260929 18:40:43.455374 13712 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 18:40:43.459585 29856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:43.456002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44776 (local address 127.13.100.62:34215)
0929 18:40:43.456128 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0929 18:40:43.456131 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:40:43.456346 (+ 215us) server_negotiation.cc:407] Connection header received
0929 18:40:43.456499 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:40:43.456502 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:40:43.456590 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:40:43.456671 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:40:43.456868 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:43.457429 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:43.458467 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:43.458692 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:43.458797 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:40:43.459104 (+ 307us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:40:43.459216 (+ 112us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0929 18:40:43.459454 (+ 238us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:44776: token expired
Metrics: {"server-negotiator.queue_time_us":48}
W20260929 18:40:43.459671 29856 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44776: token expired
W20260929 18:40:43.459936 29592 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.13.100.62:34215)
I20260929 18:40:43.467002 29856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:43.460280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44778 (local address 127.13.100.62:34215)
0929 18:40:43.460384 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0929 18:40:43.460388 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:40:43.460473 (+ 85us) server_negotiation.cc:407] Connection header received
0929 18:40:43.460601 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:40:43.460604 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:40:43.460640 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:40:43.460725 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:40:43.461305 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:43.461774 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:43.462566 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:43.462781 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:43.462900 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:40:43.464167 (+ 1267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:40:43.464170 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:40:43.464173 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:40:43.464196 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:40:43.465399 (+ 1203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:40:43.465795 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:40:43.465799 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:40:43.465801 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:40:43.465932 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:40:43.466218 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:40:43.466221 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:40:43.466223 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:40:43.466388 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:40:43.466519 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0929 18:40:43.466736 (+ 217us) server_negotiation.cc:299] Negotiation successful
0929 18:40:43.466843 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 18:40:43.469328 29666 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44778:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:40:43.472714 29666 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:40:43.480261 29788 tablet_service.cc:1467] Processing CreateTablet for tablet 9b9dac414f55412ba253d2c11113efee (DEFAULT_TABLE table=test-table [id=3ecffd7e19944aceb8a7bf1166676f03]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:40:43.480898 29788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b9dac414f55412ba253d2c11113efee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:40:43.483775 29746 tablet_bootstrap.cc:492] T 9b9dac414f55412ba253d2c11113efee P dfa07a7f1fec49beb458bd72df7589e7: Bootstrap starting.
I20260929 18:40:43.484359 30133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:43.479525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.2:36851 (local address 127.0.0.1:41216)
0929 18:40:43.479809 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0929 18:40:43.479823 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:40:43.479918 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:40:43.480603 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:40:43.480606 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:40:43.480623 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:40:43.480924 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:40:43.480930 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:40:43.482061 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:40:43.482067 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:40:43.482988 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:40:43.482998 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:40:43.483939 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:40:43.483943 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:40:43.484026 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:40:43.484077 (+ 51us) client_negotiation.cc:769] Sending connection context
0929 18:40:43.484145 (+ 68us) client_negotiation.cc:240] Negotiation successful
0929 18:40:43.484210 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"mutex_wait_us":25,"spinlock_wait_cycles":1280,"thread_start_us":104,"threads_started":1}
I20260929 18:40:43.484686 29746 tablet_bootstrap.cc:654] T 9b9dac414f55412ba253d2c11113efee P dfa07a7f1fec49beb458bd72df7589e7: Neither blocks nor log segments found. Creating new log.
I20260929 18:40:43.484874 30134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:43.479671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50090 (local address 127.13.100.3:46121)
0929 18:40:43.480143 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0929 18:40:43.480156 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 18:40:43.480179 (+ 23us) server_negotiation.cc:407] Connection header received
0929 18:40:43.480252 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:40:43.480257 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:40:43.480470 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:40:43.480800 (+ 330us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:40:43.480969 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:43.482068 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:43.483505 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:43.484229 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:43.484352 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:40:43.484502 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0929 18:40:43.484615 (+ 113us) server_negotiation.cc:299] Negotiation successful
0929 18:40:43.484716 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":98,"threads_started":1}
I20260929 18:40:43.484756 29861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:43.479244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.3:46121 (local address 127.0.0.1:50090)
0929 18:40:43.479398 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0929 18:40:43.479413 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 18:40:43.479489 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:40:43.480626 (+ 1137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:40:43.480629 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:40:43.480660 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:40:43.480810 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:40:43.480818 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:40:43.482203 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:40:43.482206 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:40:43.483345 (+ 1139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:40:43.483356 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:40:43.484352 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:40:43.484355 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:40:43.484456 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:40:43.484476 (+ 20us) client_negotiation.cc:769] Sending connection context
0929 18:40:43.484562 (+ 86us) client_negotiation.cc:240] Negotiation successful
0929 18:40:43.484637 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89,"mutex_wait_us":18,"spinlock_wait_cycles":5504}
I20260929 18:40:43.485769 30135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:43.479731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41216 (local address 127.13.100.2:36851)
0929 18:40:43.480161 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0929 18:40:43.480169 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 18:40:43.480186 (+ 17us) server_negotiation.cc:407] Connection header received
0929 18:40:43.480246 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:40:43.480250 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:40:43.480442 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:40:43.480614 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:40:43.481068 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:43.481931 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:43.483122 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:43.483714 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:43.484863 (+ 1149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:40:43.485441 (+ 578us) server_negotiation.cc:1035] Waiting for connection context
0929 18:40:43.485498 (+ 57us) server_negotiation.cc:299] Negotiation successful
0929 18:40:43.485612 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":125,"threads_started":1}
I20260929 18:40:43.486369 29746 tablet_bootstrap.cc:492] T 9b9dac414f55412ba253d2c11113efee P dfa07a7f1fec49beb458bd72df7589e7: No bootstrap required, opened a new log
I20260929 18:40:43.486455 29746 ts_tablet_manager.cc:1397] T 9b9dac414f55412ba253d2c11113efee P dfa07a7f1fec49beb458bd72df7589e7: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 18:40:43.486871 29746 raft_consensus.cc:348] T 9b9dac414f55412ba253d2c11113efee P dfa07a7f1fec49beb458bd72df7589e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f8af5f5a4024455869ea7a896d2364f" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 46121 } } peers { permanent_uuid: "248d7dfbf266486b99f1abfc09afcb8e" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 36851 } } peers { permanent_uuid: "dfa07a7f1fec49beb458bd72df7589e7" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 36845 } }
I20260929 18:40:43.486976 29746 raft_consensus.cc:374] T 9b9dac414f55412ba253d2c11113efee P dfa07a7f1fec49beb458bd72df7589e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:40:43.487025 29746 raft_consensus.cc:729] T 9b9dac414f55412ba253d2c11113efee P dfa07a7f1fec49beb458bd72df7589e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfa07a7f1fec49beb458bd72df7589e7, State: Initialized, Role: FOLLOWER
I20260929 18:40:43.487149 29746 consensus_queue.cc:260] T 9b9dac414f55412ba253d2c11113efee P dfa07a7f1fec49beb458bd72df7589e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f8af5f5a4024455869ea7a896d2364f" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 46121 } } peers { permanent_uuid: "248d7dfbf266486b99f1abfc09afcb8e" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 36851 } } peers { permanent_uuid: "dfa07a7f1fec49beb458bd72df7589e7" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 36845 } }
I20260929 18:40:43.487816 29746 ts_tablet_manager.cc:1428] T 9b9dac414f55412ba253d2c11113efee P dfa07a7f1fec49beb458bd72df7589e7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 18:40:43.488458 30058 tablet_service.cc:1467] Processing CreateTablet for tablet 9b9dac414f55412ba253d2c11113efee (DEFAULT_TABLE table=test-table [id=3ecffd7e19944aceb8a7bf1166676f03]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:40:43.488463 29926 tablet_service.cc:1467] Processing CreateTablet for tablet 9b9dac414f55412ba253d2c11113efee (DEFAULT_TABLE table=test-table [id=3ecffd7e19944aceb8a7bf1166676f03]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:40:43.489472 29926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b9dac414f55412ba253d2c11113efee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:40:43.489477 30058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b9dac414f55412ba253d2c11113efee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:40:43.495935 30137 tablet_bootstrap.cc:492] T 9b9dac414f55412ba253d2c11113efee P 248d7dfbf266486b99f1abfc09afcb8e: Bootstrap starting.
I20260929 18:40:43.495956 30138 tablet_bootstrap.cc:492] T 9b9dac414f55412ba253d2c11113efee P 0f8af5f5a4024455869ea7a896d2364f: Bootstrap starting.
I20260929 18:40:43.498178 30137 tablet_bootstrap.cc:654] T 9b9dac414f55412ba253d2c11113efee P 248d7dfbf266486b99f1abfc09afcb8e: Neither blocks nor log segments found. Creating new log.
I20260929 18:40:43.498198 30138 tablet_bootstrap.cc:654] T 9b9dac414f55412ba253d2c11113efee P 0f8af5f5a4024455869ea7a896d2364f: Neither blocks nor log segments found. Creating new log.
I20260929 18:40:43.498965 30137 log.cc:824] T 9b9dac414f55412ba253d2c11113efee P 248d7dfbf266486b99f1abfc09afcb8e: Log is configured to *not* fsync() on all Append() calls
I20260929 18:40:43.498973 30138 log.cc:824] T 9b9dac414f55412ba253d2c11113efee P 0f8af5f5a4024455869ea7a896d2364f: Log is configured to *not* fsync() on all Append() calls
I20260929 18:40:43.501073 30137 tablet_bootstrap.cc:492] T 9b9dac414f55412ba253d2c11113efee P 248d7dfbf266486b99f1abfc09afcb8e: No bootstrap required, opened a new log
I20260929 18:40:43.501084 30138 tablet_bootstrap.cc:492] T 9b9dac414f55412ba253d2c11113efee P 0f8af5f5a4024455869ea7a896d2364f: No bootstrap required, opened a new log
I20260929 18:40:43.501266 30137 ts_tablet_manager.cc:1397] T 9b9dac414f55412ba253d2c11113efee P 248d7dfbf266486b99f1abfc09afcb8e: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260929 18:40:43.501266 30138 ts_tablet_manager.cc:1397] T 9b9dac414f55412ba253d2c11113efee P 0f8af5f5a4024455869ea7a896d2364f: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 18:40:43.505146 30137 raft_consensus.cc:348] T 9b9dac414f55412ba253d2c11113efee P 248d7dfbf266486b99f1abfc09afcb8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f8af5f5a4024455869ea7a896d2364f" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 46121 } } peers { permanent_uuid: "248d7dfbf266486b99f1abfc09afcb8e" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 36851 } } peers { permanent_uuid: "dfa07a7f1fec49beb458bd72df7589e7" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 36845 } }
I20260929 18:40:43.505146 30138 raft_consensus.cc:348] T 9b9dac414f55412ba253d2c11113efee P 0f8af5f5a4024455869ea7a896d2364f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f8af5f5a4024455869ea7a896d2364f" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 46121 } } peers { permanent_uuid: "248d7dfbf266486b99f1abfc09afcb8e" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 36851 } } peers { permanent_uuid: "dfa07a7f1fec49beb458bd72df7589e7" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 36845 } }
I20260929 18:40:43.505432 30138 raft_consensus.cc:374] T 9b9dac414f55412ba253d2c11113efee P 0f8af5f5a4024455869ea7a896d2364f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:40:43.505432 30137 raft_consensus.cc:374] T 9b9dac414f55412ba253d2c11113efee P 248d7dfbf266486b99f1abfc09afcb8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:40:43.505494 30138 raft_consensus.cc:729] T 9b9dac414f55412ba253d2c11113efee P 0f8af5f5a4024455869ea7a896d2364f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f8af5f5a4024455869ea7a896d2364f, State: Initialized, Role: FOLLOWER
I20260929 18:40:43.505494 30137 raft_consensus.cc:729] T 9b9dac414f55412ba253d2c11113efee P 248d7dfbf266486b99f1abfc09afcb8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 248d7dfbf266486b99f1abfc09afcb8e, State: Initialized, Role: FOLLOWER
I20260929 18:40:43.505982 30138 consensus_queue.cc:260] T 9b9dac414f55412ba253d2c11113efee P 0f8af5f5a4024455869ea7a896d2364f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f8af5f5a4024455869ea7a896d2364f" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 46121 } } peers { permanent_uuid: "248d7dfbf266486b99f1abfc09afcb8e" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 36851 } } peers { permanent_uuid: "dfa07a7f1fec49beb458bd72df7589e7" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 36845 } }
I20260929 18:40:43.505982 30137 consensus_queue.cc:260] T 9b9dac414f55412ba253d2c11113efee P 248d7dfbf266486b99f1abfc09afcb8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f8af5f5a4024455869ea7a896d2364f" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 46121 } } peers { permanent_uuid: "248d7dfbf266486b99f1abfc09afcb8e" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 36851 } } peers { permanent_uuid: "dfa07a7f1fec49beb458bd72df7589e7" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 36845 } }
I20260929 18:40:43.506675 29992 heartbeater.cc:503] Master 127.13.100.62:34215 was elected leader, sending a full tablet report...
I20260929 18:40:43.507000 30138 ts_tablet_manager.cc:1428] T 9b9dac414f55412ba253d2c11113efee P 0f8af5f5a4024455869ea7a896d2364f: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260929 18:40:43.507000 30137 ts_tablet_manager.cc:1428] T 9b9dac414f55412ba253d2c11113efee P 248d7dfbf266486b99f1abfc09afcb8e: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 18:40:43.507699 30124 heartbeater.cc:503] Master 127.13.100.62:34215 was elected leader, sending a full tablet report...
W20260929 18:40:43.522879 29855 tablet.cc:2369] T 9b9dac414f55412ba253d2c11113efee P dfa07a7f1fec49beb458bd72df7589e7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 18:40:43.680786 30125 tablet.cc:2369] T 9b9dac414f55412ba253d2c11113efee P 0f8af5f5a4024455869ea7a896d2364f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 18:40:43.730798 29993 tablet.cc:2369] T 9b9dac414f55412ba253d2c11113efee P 248d7dfbf266486b99f1abfc09afcb8e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:40:43.772653 30141 raft_consensus.cc:482] T 9b9dac414f55412ba253d2c11113efee P 248d7dfbf266486b99f1abfc09afcb8e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:40:43.772876 30141 raft_consensus.cc:504] T 9b9dac414f55412ba253d2c11113efee P 248d7dfbf266486b99f1abfc09afcb8e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f8af5f5a4024455869ea7a896d2364f" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 46121 } } peers { permanent_uuid: "248d7dfbf266486b99f1abfc09afcb8e" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 36851 } } peers { permanent_uuid: "dfa07a7f1fec49beb458bd72df7589e7" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 36845 } }
I20260929 18:40:43.774112 30141 leader_election.cc:288] T 9b9dac414f55412ba253d2c11113efee P 248d7dfbf266486b99f1abfc09afcb8e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0f8af5f5a4024455869ea7a896d2364f (127.13.100.3:46121), dfa07a7f1fec49beb458bd72df7589e7 (127.13.100.1:36845)
I20260929 18:40:43.778084 30145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:43.774484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.2:40477 (local address 127.13.100.1:36845)
0929 18:40:43.774774 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0929 18:40:43.774780 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:40:43.774797 (+ 17us) server_negotiation.cc:407] Connection header received
0929 18:40:43.774961 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:40:43.774964 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:40:43.775070 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:40:43.775174 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:40:43.775446 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:43.776006 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:43.776991 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:43.777410 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:43.777549 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:40:43.777722 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0929 18:40:43.777831 (+ 109us) server_negotiation.cc:299] Negotiation successful
0929 18:40:43.777930 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":217,"thread_start_us":95,"threads_started":1}
I20260929 18:40:43.778084 30134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:43.774424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.2:45167 (local address 127.13.100.3:46121)
0929 18:40:43.774569 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0929 18:40:43.774574 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:40:43.774708 (+ 134us) server_negotiation.cc:407] Connection header received
0929 18:40:43.774894 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:40:43.774899 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:40:43.774947 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:40:43.775026 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:40:43.775307 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:43.775824 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:43.776957 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:43.777425 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:43.777559 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:40:43.777612 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0929 18:40:43.777849 (+ 237us) server_negotiation.cc:299] Negotiation successful
0929 18:40:43.777954 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 18:40:43.778105 30143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:43.774363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.1:36845 (local address 127.13.100.2:40477)
0929 18:40:43.774734 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0929 18:40:43.774748 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:40:43.774844 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:40:43.775193 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:40:43.775196 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:40:43.775212 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:40:43.775337 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:40:43.775343 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:40:43.776114 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:40:43.776118 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:40:43.776861 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:40:43.776869 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:40:43.777538 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:40:43.777542 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:40:43.777630 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:40:43.777736 (+ 106us) client_negotiation.cc:769] Sending connection context
0929 18:40:43.777806 (+ 70us) client_negotiation.cc:240] Negotiation successful
0929 18:40:43.777946 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"mutex_wait_us":59,"thread_start_us":64,"threads_started":1}
I20260929 18:40:43.778465 30144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:43.774287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.3:46121 (local address 127.13.100.2:45167)
0929 18:40:43.774605 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0929 18:40:43.774623 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 18:40:43.774787 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:40:43.775046 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:40:43.775048 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:40:43.775064 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:40:43.775189 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:40:43.775212 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:40:43.775935 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:40:43.775940 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:40:43.776808 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:40:43.776817 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:40:43.777549 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:40:43.777553 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:40:43.777732 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:40:43.777750 (+ 18us) client_negotiation.cc:769] Sending connection context
0929 18:40:43.777833 (+ 83us) client_negotiation.cc:240] Negotiation successful
0929 18:40:43.778067 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"mutex_wait_us":184,"thread_start_us":102,"threads_started":1}
I20260929 18:40:43.778627 29808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b9dac414f55412ba253d2c11113efee" candidate_uuid: "248d7dfbf266486b99f1abfc09afcb8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfa07a7f1fec49beb458bd72df7589e7" is_pre_election: true
I20260929 18:40:43.778937 29808 raft_consensus.cc:2454] T 9b9dac414f55412ba253d2c11113efee P dfa07a7f1fec49beb458bd72df7589e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 248d7dfbf266486b99f1abfc09afcb8e in term 0.
I20260929 18:40:43.779047 30078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b9dac414f55412ba253d2c11113efee" candidate_uuid: "248d7dfbf266486b99f1abfc09afcb8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f8af5f5a4024455869ea7a896d2364f" is_pre_election: true
I20260929 18:40:43.779311 30078 raft_consensus.cc:2454] T 9b9dac414f55412ba253d2c11113efee P 0f8af5f5a4024455869ea7a896d2364f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 248d7dfbf266486b99f1abfc09afcb8e in term 0.
I20260929 18:40:43.779421 29880 leader_election.cc:302] T 9b9dac414f55412ba253d2c11113efee P 248d7dfbf266486b99f1abfc09afcb8e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 248d7dfbf266486b99f1abfc09afcb8e, dfa07a7f1fec49beb458bd72df7589e7; no voters:
I20260929 18:40:43.779703 30141 raft_consensus.cc:2781] T 9b9dac414f55412ba253d2c11113efee P 248d7dfbf266486b99f1abfc09afcb8e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:40:43.779794 30141 raft_consensus.cc:482] T 9b9dac414f55412ba253d2c11113efee P 248d7dfbf266486b99f1abfc09afcb8e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:40:43.779838 30141 raft_consensus.cc:3037] T 9b9dac414f55412ba253d2c11113efee P 248d7dfbf266486b99f1abfc09afcb8e [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:40:43.780818 30141 raft_consensus.cc:504] T 9b9dac414f55412ba253d2c11113efee P 248d7dfbf266486b99f1abfc09afcb8e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f8af5f5a4024455869ea7a896d2364f" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 46121 } } peers { permanent_uuid: "248d7dfbf266486b99f1abfc09afcb8e" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 36851 } } peers { permanent_uuid: "dfa07a7f1fec49beb458bd72df7589e7" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 36845 } }
I20260929 18:40:43.781148 30141 leader_election.cc:288] T 9b9dac414f55412ba253d2c11113efee P 248d7dfbf266486b99f1abfc09afcb8e [CANDIDATE]: Term 1 election: Requested vote from peers 0f8af5f5a4024455869ea7a896d2364f (127.13.100.3:46121), dfa07a7f1fec49beb458bd72df7589e7 (127.13.100.1:36845)
I20260929 18:40:43.781424 30078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b9dac414f55412ba253d2c11113efee" candidate_uuid: "248d7dfbf266486b99f1abfc09afcb8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f8af5f5a4024455869ea7a896d2364f"
I20260929 18:40:43.781570 30078 raft_consensus.cc:3037] T 9b9dac414f55412ba253d2c11113efee P 0f8af5f5a4024455869ea7a896d2364f [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:40:43.781584 29808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b9dac414f55412ba253d2c11113efee" candidate_uuid: "248d7dfbf266486b99f1abfc09afcb8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfa07a7f1fec49beb458bd72df7589e7"
I20260929 18:40:43.781705 29808 raft_consensus.cc:3037] T 9b9dac414f55412ba253d2c11113efee P dfa07a7f1fec49beb458bd72df7589e7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:40:43.782837 30078 raft_consensus.cc:2454] T 9b9dac414f55412ba253d2c11113efee P 0f8af5f5a4024455869ea7a896d2364f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 248d7dfbf266486b99f1abfc09afcb8e in term 1.
I20260929 18:40:43.782837 29808 raft_consensus.cc:2454] T 9b9dac414f55412ba253d2c11113efee P dfa07a7f1fec49beb458bd72df7589e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 248d7dfbf266486b99f1abfc09afcb8e in term 1.
I20260929 18:40:43.783123 29880 leader_election.cc:302] T 9b9dac414f55412ba253d2c11113efee P 248d7dfbf266486b99f1abfc09afcb8e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0f8af5f5a4024455869ea7a896d2364f, 248d7dfbf266486b99f1abfc09afcb8e, dfa07a7f1fec49beb458bd72df7589e7; no voters:
I20260929 18:40:43.783270 30141 raft_consensus.cc:2781] T 9b9dac414f55412ba253d2c11113efee P 248d7dfbf266486b99f1abfc09afcb8e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:40:43.783489 30141 raft_consensus.cc:686] T 9b9dac414f55412ba253d2c11113efee P 248d7dfbf266486b99f1abfc09afcb8e [term 1 LEADER]: Becoming Leader. State: Replica: 248d7dfbf266486b99f1abfc09afcb8e, State: Running, Role: LEADER
I20260929 18:40:43.783839 30141 consensus_queue.cc:237] T 9b9dac414f55412ba253d2c11113efee P 248d7dfbf266486b99f1abfc09afcb8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f8af5f5a4024455869ea7a896d2364f" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 46121 } } peers { permanent_uuid: "248d7dfbf266486b99f1abfc09afcb8e" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 36851 } } peers { permanent_uuid: "dfa07a7f1fec49beb458bd72df7589e7" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 36845 } }
I20260929 18:40:43.786967 29666 catalog_manager.cc:5495] T 9b9dac414f55412ba253d2c11113efee P 248d7dfbf266486b99f1abfc09afcb8e reported cstate change: term changed from 0 to 1, leader changed from <none> to 248d7dfbf266486b99f1abfc09afcb8e (127.13.100.2). New cstate: current_term: 1 leader_uuid: "248d7dfbf266486b99f1abfc09afcb8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f8af5f5a4024455869ea7a896d2364f" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 46121 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "248d7dfbf266486b99f1abfc09afcb8e" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 36851 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dfa07a7f1fec49beb458bd72df7589e7" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 36845 } health_report { overall_health: UNKNOWN } } }
I20260929 18:40:43.850255 30135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:43.847015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41222 (local address 127.13.100.2:36851)
0929 18:40:43.847151 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0929 18:40:43.847155 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:40:43.847169 (+ 14us) server_negotiation.cc:407] Connection header received
0929 18:40:43.847284 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:40:43.847287 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:40:43.847340 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:40:43.847465 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:40:43.847650 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:43.848152 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:43.848932 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:40:43.849142 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:40:43.849225 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:40:43.849485 (+ 260us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:40:43.849873 (+ 388us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:40:43.849981 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 18:40:43.850063 (+ 82us) server_negotiation.cc:299] Negotiation successful
0929 18:40:43.850109 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 18:40:43.855913 29808 raft_consensus.cc:1261] T 9b9dac414f55412ba253d2c11113efee P dfa07a7f1fec49beb458bd72df7589e7 [term 1 FOLLOWER]: Refusing update from remote peer 248d7dfbf266486b99f1abfc09afcb8e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:40:43.855957 30078 raft_consensus.cc:1261] T 9b9dac414f55412ba253d2c11113efee P 0f8af5f5a4024455869ea7a896d2364f [term 1 FOLLOWER]: Refusing update from remote peer 248d7dfbf266486b99f1abfc09afcb8e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:40:43.856637 30146 consensus_queue.cc:1035] T 9b9dac414f55412ba253d2c11113efee P 248d7dfbf266486b99f1abfc09afcb8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa07a7f1fec49beb458bd72df7589e7" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 36845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:40:43.856869 30141 consensus_queue.cc:1035] T 9b9dac414f55412ba253d2c11113efee P 248d7dfbf266486b99f1abfc09afcb8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f8af5f5a4024455869ea7a896d2364f" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 46121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:40:43.863350 30149 mvcc.cc:205] Tried to move back new op lower bound from 7334736870826737664 to 7334736870541455360. Current Snapshot: MvccSnapshot[applied={T|T < 7334736870826737664}]
I20260929 18:40:43.866144 30150 mvcc.cc:205] Tried to move back new op lower bound from 7334736870826737664 to 7334736870541455360. Current Snapshot: MvccSnapshot[applied={T|T < 7334736870826737664}]
I20260929 18:40:43.871551 29666 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44778:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:40:43.871718 29666 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44778:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:40:43.874236 29666 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 55348a079a5f4d1795e48eda404bb105: Sending DeleteTablet for 3 replicas of tablet 9b9dac414f55412ba253d2c11113efee
I20260929 18:40:43.875005 30058 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b9dac414f55412ba253d2c11113efee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:40:43 UTC) from {username='kudu', principal='kudu/127.13.100.62@KRBTEST.COM'} at 127.0.0.1:50090
I20260929 18:40:43.874991 29926 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b9dac414f55412ba253d2c11113efee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:40:43 UTC) from {username='kudu', principal='kudu/127.13.100.62@KRBTEST.COM'} at 127.0.0.1:41216
I20260929 18:40:43.875442 29788 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b9dac414f55412ba253d2c11113efee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:40:43 UTC) from {username='kudu', principal='kudu/127.13.100.62@KRBTEST.COM'} at 127.0.0.1:40478
I20260929 18:40:43.875590 30159 tablet_replica.cc:330] T 9b9dac414f55412ba253d2c11113efee P 0f8af5f5a4024455869ea7a896d2364f: stopping tablet replica
I20260929 18:40:43.875849 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 29704
I20260929 18:40:43.875962 30159 raft_consensus.cc:2229] T 9b9dac414f55412ba253d2c11113efee P 0f8af5f5a4024455869ea7a896d2364f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:40:43.876268 30159 raft_consensus.cc:2258] T 9b9dac414f55412ba253d2c11113efee P 0f8af5f5a4024455869ea7a896d2364f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:40:43.877815 30159 ts_tablet_manager.cc:1905] T 9b9dac414f55412ba253d2c11113efee P 0f8af5f5a4024455869ea7a896d2364f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:40:43.881295 30159 ts_tablet_manager.cc:1918] T 9b9dac414f55412ba253d2c11113efee P 0f8af5f5a4024455869ea7a896d2364f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:40:43.881413 30159 log.cc:1196] T 9b9dac414f55412ba253d2c11113efee P 0f8af5f5a4024455869ea7a896d2364f: Deleting WAL directory at /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/ts-2/wal/wals/9b9dac414f55412ba253d2c11113efee
I20260929 18:40:43.881747 30160 tablet_replica.cc:330] T 9b9dac414f55412ba253d2c11113efee P 248d7dfbf266486b99f1abfc09afcb8e: stopping tablet replica
I20260929 18:40:43.881776 30159 ts_tablet_manager.cc:1939] T 9b9dac414f55412ba253d2c11113efee P 0f8af5f5a4024455869ea7a896d2364f: Deleting consensus metadata
I20260929 18:40:43.882205 30160 raft_consensus.cc:2229] T 9b9dac414f55412ba253d2c11113efee P 248d7dfbf266486b99f1abfc09afcb8e [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:40:43.882624 30160 raft_consensus.cc:2258] T 9b9dac414f55412ba253d2c11113efee P 248d7dfbf266486b99f1abfc09afcb8e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:40:43.883208 29650 catalog_manager.cc:4841] TS 0f8af5f5a4024455869ea7a896d2364f (127.13.100.3:46121): tablet 9b9dac414f55412ba253d2c11113efee (table test-table [id=3ecffd7e19944aceb8a7bf1166676f03]) successfully deleted
I20260929 18:40:43.884284 30160 ts_tablet_manager.cc:1905] T 9b9dac414f55412ba253d2c11113efee P 248d7dfbf266486b99f1abfc09afcb8e: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260929 18:40:43.884572 29651 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.100.1:36845 (error 108)
W20260929 18:40:43.885653 30133 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:40:43.885273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.1:36845 (local address 127.0.0.1:40494)
0929 18:40:43.885481 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0929 18:40:43.885552 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.1:36845: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":103}
I20260929 18:40:43.885710 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 29858
W20260929 18:40:43.885949 29651 catalog_manager.cc:4584] TS dfa07a7f1fec49beb458bd72df7589e7 (127.13.100.1:36845): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9b9dac414f55412ba253d2c11113efee: Network error: Client connection negotiation failed: client connection to 127.13.100.1:36845: connect: Connection reset by peer (error 104)
I20260929 18:40:43.892227 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 29995
I20260929 18:40:43.892460 30133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:40:43.892173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.2:36851 (local address 127.0.0.1:41234)
0929 18:40:43.892308 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0929 18:40:43.892370 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.2:36851: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260929 18:40:43.897804 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 29633
I20260929 18:41:03.904033 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.100.62:34215
--webserver_interface=127.13.100.62
--webserver_port=42543
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.62.keytab
--principal=kudu/127.13.100.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.100.20:44697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.100.62:34215
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:41:04.014336 30163 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:41:04.014599 30163 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:41:04.014640 30163 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:41:04.018625 30163 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 18:41:04.018677 30163 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 18:41:04.018692 30163 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 18:41:04.018707 30163 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:41:04.018724 30163 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:41:04.018742 30163 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:41:04.018807 30163 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:41:04.018839 30163 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:41:04.024158 30163 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.100.20:44697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.13.100.62:34215
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.62.keytab
--principal=kudu/127.13.100.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.100.62:34215
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.100.62
--webserver_port=42543
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30163
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:41:04.025243 30163 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:41:04.026299 30163 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:41:04.035310 30171 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:41:04.035377 30163 server_base.cc:1034] running on GCE node
W20260929 18:41:04.035329 30169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:41:04.035317 30168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:41:04.036222 30163 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:41:04.037318 30163 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:41:04.038529 30163 hybrid_clock.cc:648] HybridClock initialized: now 1790707264038488 us; error 49 us; skew 500 ppm
Sep 29 18:41:04 dist-test-slave-j1xs krb5kdc[29084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707264, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:41:04.041308 30163 init.cc:377] Logged in from keytab as kudu/127.13.100.62@KRBTEST.COM (short username kudu)
I20260929 18:41:04.042263 30163 webserver.cc:460] Webserver started at http://127.13.100.62:42543/ using document root <none> and password file <none>
I20260929 18:41:04.042763 30163 fs_manager.cc:359] Metadata directory not provided
I20260929 18:41:04.042870 30163 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:41:04.046687 30163 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:41:04.048712 30178 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:41:04.050035 30163 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:41:04.050154 30163 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/master-0/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/master-0/wal
uuid: "55348a079a5f4d1795e48eda404bb105"
format_stamp: "Formatted at 2026-09-29 18:40:21 on dist-test-slave-j1xs"
server_key: "3028baaa3654086d63191a66f22284da"
server_key_iv: "f057f747b98d8b08403bbef039e93785"
server_key_version: "encryptionkey@0"
I20260929 18:41:04.050926 30163 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:41:04.065239 30163 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:41:04.068490 30163 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:41:04.068704 30163 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:41:04.076197 30163 rpc_server.cc:307] RPC server started. Bound to: 127.13.100.62:34215
I20260929 18:41:04.076469 30230 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.100.62:34215 every 8 connection(s)
I20260929 18:41:04.077354 30163 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
I20260929 18:41:04.080561 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 30163
I20260929 18:41:04.080992 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.100.1:36845
--local_ip_for_outbound_sockets=127.13.100.1
--tserver_master_addrs=127.13.100.62:34215
--webserver_port=38303
--webserver_interface=127.13.100.1
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.1.keytab
--principal=kudu/127.13.100.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.100.20:44697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 18:41:04.081683 30231 sys_catalog.cc:263] Verifying existing consensus state
I20260929 18:41:04.083331 30231 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55348a079a5f4d1795e48eda404bb105: Bootstrap starting.
I20260929 18:41:04.100425 30231 log.cc:824] T 00000000000000000000000000000000 P 55348a079a5f4d1795e48eda404bb105: Log is configured to *not* fsync() on all Append() calls
I20260929 18:41:04.111423 30231 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55348a079a5f4d1795e48eda404bb105: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 18:41:04.111817 30231 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55348a079a5f4d1795e48eda404bb105: Bootstrap complete.
I20260929 18:41:04.115772 30231 raft_consensus.cc:348] T 00000000000000000000000000000000 P 55348a079a5f4d1795e48eda404bb105 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55348a079a5f4d1795e48eda404bb105" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 34215 } }
I20260929 18:41:04.116145 30231 raft_consensus.cc:729] T 00000000000000000000000000000000 P 55348a079a5f4d1795e48eda404bb105 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55348a079a5f4d1795e48eda404bb105, State: Initialized, Role: FOLLOWER
I20260929 18:41:04.116616 30231 consensus_queue.cc:260] T 00000000000000000000000000000000 P 55348a079a5f4d1795e48eda404bb105 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55348a079a5f4d1795e48eda404bb105" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 34215 } }
I20260929 18:41:04.116724 30231 raft_consensus.cc:388] T 00000000000000000000000000000000 P 55348a079a5f4d1795e48eda404bb105 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:41:04.116814 30231 raft_consensus.cc:482] T 00000000000000000000000000000000 P 55348a079a5f4d1795e48eda404bb105 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:41:04.116925 30231 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 55348a079a5f4d1795e48eda404bb105 [term 2 FOLLOWER]: Advancing to term 3
I20260929 18:41:04.118824 30231 raft_consensus.cc:504] T 00000000000000000000000000000000 P 55348a079a5f4d1795e48eda404bb105 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55348a079a5f4d1795e48eda404bb105" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 34215 } }
I20260929 18:41:04.119163 30231 leader_election.cc:302] T 00000000000000000000000000000000 P 55348a079a5f4d1795e48eda404bb105 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 55348a079a5f4d1795e48eda404bb105; no voters:
I20260929 18:41:04.119480 30231 leader_election.cc:288] T 00000000000000000000000000000000 P 55348a079a5f4d1795e48eda404bb105 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260929 18:41:04.119588 30235 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 55348a079a5f4d1795e48eda404bb105 [term 3 FOLLOWER]: Leader election won for term 3
I20260929 18:41:04.119787 30235 raft_consensus.cc:686] T 00000000000000000000000000000000 P 55348a079a5f4d1795e48eda404bb105 [term 3 LEADER]: Becoming Leader. State: Replica: 55348a079a5f4d1795e48eda404bb105, State: Running, Role: LEADER
I20260929 18:41:04.120127 30235 consensus_queue.cc:237] T 00000000000000000000000000000000 P 55348a079a5f4d1795e48eda404bb105 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55348a079a5f4d1795e48eda404bb105" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 34215 } }
I20260929 18:41:04.120671 30231 sys_catalog.cc:564] T 00000000000000000000000000000000 P 55348a079a5f4d1795e48eda404bb105 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:41:04.121006 30235 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55348a079a5f4d1795e48eda404bb105 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 55348a079a5f4d1795e48eda404bb105. Latest consensus state: current_term: 3 leader_uuid: "55348a079a5f4d1795e48eda404bb105" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55348a079a5f4d1795e48eda404bb105" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 34215 } } }
I20260929 18:41:04.121202 30235 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55348a079a5f4d1795e48eda404bb105 [sys.catalog]: This master's current role is: LEADER
I20260929 18:41:04.121693 30236 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55348a079a5f4d1795e48eda404bb105 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "55348a079a5f4d1795e48eda404bb105" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55348a079a5f4d1795e48eda404bb105" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 34215 } } }
I20260929 18:41:04.121865 30236 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55348a079a5f4d1795e48eda404bb105 [sys.catalog]: This master's current role is: LEADER
I20260929 18:41:04.121939 30239 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:41:04.126999 30239 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:41:04.127797 30239 catalog_manager.cc:1259] Loaded cluster ID: e62d58bc40c54a52be38c7f6f3340271
I20260929 18:41:04.127861 30239 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:41:04.130366 30239 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:41:04.131145 30239 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 55348a079a5f4d1795e48eda404bb105: Loaded TSK: 0
I20260929 18:41:04.131225 30239 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 55348a079a5f4d1795e48eda404bb105: Loaded TSK: 1
F20260929 18:41:04.131980 30239 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7ff62a92a9ad google::LogMessage::Fail() at ??:0
@ 0x7ff62a92d88a google::LogMessage::SendToLog() at ??:0
@ 0x7ff62a92a4de google::LogMessage::Flush() at ??:0
@ 0x7ff62a92ba19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7ff62d5ee18d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7ff62d5e6f0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7ff63528f7bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7ff63526999c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7ff63526a178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7ff6352abdf8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7ff6352ca82b std::function<>::operator()() at ??:0
@ 0x7ff635269b4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7ff63526af3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7ff635267579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7ff6352ab640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7ff62cb7c3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7ff62cb7cc83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7ff62cb7e3f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7ff62cb6d512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7ff62e1b16db start_thread at ??:0
@ 0x7ff62942861f clone at ??:0
W20260929 18:41:04.221846 30233 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:41:04.222222 30233 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:41:04.222295 30233 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:41:04.228819 30233 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:41:04.228886 30233 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:41:04.228986 30233 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.100.1
I20260929 18:41:04.237458 30233 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.100.20:44697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.1.keytab
--principal=kudu/127.13.100.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.100.1:36845
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.100.1
--webserver_port=38303
--tserver_master_addrs=127.13.100.62:34215
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30233
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.100.1
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:41:04.238855 30233 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:41:04.240190 30233 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:41:04.253165 30279 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:41:04.253351 30233 server_base.cc:1034] running on GCE node
W20260929 18:41:04.253206 30278 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:41:04.253360 30281 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:41:04.254067 30233 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:41:04.254760 30233 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:41:04.255945 30233 hybrid_clock.cc:648] HybridClock initialized: now 1790707264255930 us; error 33 us; skew 500 ppm
Sep 29 18:41:04 dist-test-slave-j1xs krb5kdc[29084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707264, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:41:04.259352 30233 init.cc:377] Logged in from keytab as kudu/127.13.100.1@KRBTEST.COM (short username kudu)
I20260929 18:41:04.260462 30233 webserver.cc:460] Webserver started at http://127.13.100.1:38303/ using document root <none> and password file <none>
I20260929 18:41:04.261168 30233 fs_manager.cc:359] Metadata directory not provided
I20260929 18:41:04.261268 30233 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:41:04.266209 30233 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 18:41:04.268862 30288 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:41:04.270406 30233 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:41:04.270541 30233 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/ts-0/wal
uuid: "dfa07a7f1fec49beb458bd72df7589e7"
format_stamp: "Formatted at 2026-09-29 18:40:21 on dist-test-slave-j1xs"
server_key: "678290f32b6284b80e3321b76a46581c"
server_key_iv: "546083d1f5bf912f0549e28f3e4e8826"
server_key_version: "encryptionkey@0"
I20260929 18:41:04.271270 30233 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:41:04.293833 30233 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:41:04.298602 30233 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:41:04.298812 30233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:41:04.299597 30233 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:41:04.301092 30295 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260929 18:41:04.305501 30233 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 18:41:04.305774 30233 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260929 18:41:04.305876 30233 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 18:41:04.308583 30233 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 18:41:04.308668 30233 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.000s sys 0.000s
I20260929 18:41:04.309429 30295 tablet_bootstrap.cc:492] T 9b9dac414f55412ba253d2c11113efee P dfa07a7f1fec49beb458bd72df7589e7: Bootstrap starting.
I20260929 18:41:04.321336 30233 rpc_server.cc:307] RPC server started. Bound to: 127.13.100.1:36845
I20260929 18:41:04.321447 30402 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.100.1:36845 every 8 connection(s)
I20260929 18:41:04.322769 30233 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/ts-0/data/info.pb
I20260929 18:41:04.328449 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 30233
I20260929 18:41:04.329105 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.100.2:36851
--local_ip_for_outbound_sockets=127.13.100.2
--tserver_master_addrs=127.13.100.62:34215
--webserver_port=36695
--webserver_interface=127.13.100.2
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.2.keytab
--principal=kudu/127.13.100.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.100.20:44697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 18:41:04.331394 30295 log.cc:824] T 9b9dac414f55412ba253d2c11113efee P dfa07a7f1fec49beb458bd72df7589e7: Log is configured to *not* fsync() on all Append() calls
I20260929 18:41:04.339435 30295 tablet_bootstrap.cc:492] T 9b9dac414f55412ba253d2c11113efee P dfa07a7f1fec49beb458bd72df7589e7: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 18:41:04.339855 30295 tablet_bootstrap.cc:492] T 9b9dac414f55412ba253d2c11113efee P dfa07a7f1fec49beb458bd72df7589e7: Bootstrap complete.
I20260929 18:41:04.341091 30295 ts_tablet_manager.cc:1397] T 9b9dac414f55412ba253d2c11113efee P dfa07a7f1fec49beb458bd72df7589e7: Time spent bootstrapping tablet: real 0.032s user 0.017s sys 0.012s
I20260929 18:41:04.343576 30295 raft_consensus.cc:348] T 9b9dac414f55412ba253d2c11113efee P dfa07a7f1fec49beb458bd72df7589e7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f8af5f5a4024455869ea7a896d2364f" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 46121 } } peers { permanent_uuid: "248d7dfbf266486b99f1abfc09afcb8e" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 36851 } } peers { permanent_uuid: "dfa07a7f1fec49beb458bd72df7589e7" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 36845 } }
I20260929 18:41:04.344120 30295 raft_consensus.cc:729] T 9b9dac414f55412ba253d2c11113efee P dfa07a7f1fec49beb458bd72df7589e7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfa07a7f1fec49beb458bd72df7589e7, State: Initialized, Role: FOLLOWER
I20260929 18:41:04.344652 30295 consensus_queue.cc:260] T 9b9dac414f55412ba253d2c11113efee P dfa07a7f1fec49beb458bd72df7589e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f8af5f5a4024455869ea7a896d2364f" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 46121 } } peers { permanent_uuid: "248d7dfbf266486b99f1abfc09afcb8e" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 36851 } } peers { permanent_uuid: "dfa07a7f1fec49beb458bd72df7589e7" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 36845 } }
I20260929 18:41:04.345705 30295 ts_tablet_manager.cc:1428] T 9b9dac414f55412ba253d2c11113efee P dfa07a7f1fec49beb458bd72df7589e7: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
W20260929 18:41:04.420341 30405 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:41:04.325176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:34215 (local address 127.13.100.1:52561)
0929 18:41:04.325804 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:04.325870 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0929 18:41:04.327041 (+ 1171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:41:04.419374 (+ 92333us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.62:34215: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":140,"threads_started":1}
W20260929 18:41:04.420948 30403 heartbeater.cc:644] Failed to heartbeat to 127.13.100.62:34215 (0 consecutive failures): Network error: Failed to ping master at 127.13.100.62:34215: Client connection negotiation failed: client connection to 127.13.100.62:34215: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 18:41:04.421713 30405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:04.421412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:34215 (local address 127.13.100.1:32995)
0929 18:41:04.421556 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:04.421639 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.62:34215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260929 18:41:04.422399 30405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:04.422181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:34215 (local address 127.13.100.1:55451)
0929 18:41:04.422294 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:04.422345 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.62:34215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260929 18:41:04.422590 30403 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 18:41:04.466104 30406 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:41:04.466339 30406 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:41:04.466382 30406 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:41:04.470261 30406 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:41:04.470317 30406 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:41:04.470373 30406 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.100.2
I20260929 18:41:04.475454 30406 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.100.20:44697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.2.keytab
--principal=kudu/127.13.100.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.100.2:36851
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.100.2
--webserver_port=36695
--tserver_master_addrs=127.13.100.62:34215
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30406
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.100.2
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:41:04.476460 30406 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:41:04.477412 30406 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:41:04.486748 30414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:41:04.486756 30416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:41:04.486759 30413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:41:04.486984 30406 server_base.cc:1034] running on GCE node
I20260929 18:41:04.487419 30406 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:41:04.488073 30406 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:41:04.489243 30406 hybrid_clock.cc:648] HybridClock initialized: now 1790707264489229 us; error 27 us; skew 500 ppm
Sep 29 18:41:04 dist-test-slave-j1xs krb5kdc[29084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707264, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:41:04.492018 30406 init.cc:377] Logged in from keytab as kudu/127.13.100.2@KRBTEST.COM (short username kudu)
I20260929 18:41:04.492894 30406 webserver.cc:460] Webserver started at http://127.13.100.2:36695/ using document root <none> and password file <none>
I20260929 18:41:04.493456 30406 fs_manager.cc:359] Metadata directory not provided
I20260929 18:41:04.493561 30406 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:41:04.497371 30406 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:41:04.499599 30423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:41:04.500890 30406 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 18:41:04.501060 30406 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/ts-1/wal
uuid: "248d7dfbf266486b99f1abfc09afcb8e"
format_stamp: "Formatted at 2026-09-29 18:40:22 on dist-test-slave-j1xs"
server_key: "dcf8d37d7b3add7c0354687c0d3bfd2e"
server_key_iv: "56e823f10c16a4a9bb098bf07a198f47"
server_key_version: "encryptionkey@0"
I20260929 18:41:04.501636 30406 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:41:04.520424 30406 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:41:04.523550 30406 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:41:04.523767 30406 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:41:04.524381 30406 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:41:04.525558 30430 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260929 18:41:04.529587 30430 ts_tablet_manager.cc:1852] T 9b9dac414f55412ba253d2c11113efee P 248d7dfbf266486b99f1abfc09afcb8e: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260929 18:41:04.530035 30430 ts_tablet_manager.cc:1905] T 9b9dac414f55412ba253d2c11113efee P 248d7dfbf266486b99f1abfc09afcb8e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:41:04.534125 30430 ts_tablet_manager.cc:1918] T 9b9dac414f55412ba253d2c11113efee P 248d7dfbf266486b99f1abfc09afcb8e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:41:04.534230 30430 log.cc:1196] T 9b9dac414f55412ba253d2c11113efee P 248d7dfbf266486b99f1abfc09afcb8e: Deleting WAL directory at /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/ts-1/wal/wals/9b9dac414f55412ba253d2c11113efee
I20260929 18:41:04.534626 30430 ts_tablet_manager.cc:1939] T 9b9dac414f55412ba253d2c11113efee P 248d7dfbf266486b99f1abfc09afcb8e: Deleting consensus metadata
I20260929 18:41:04.534994 30406 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260929 18:41:04.535094 30406 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.010s user 0.000s sys 0.000s
I20260929 18:41:04.535182 30406 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:41:04.535234 30406 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:41:04.544471 30406 rpc_server.cc:307] RPC server started. Bound to: 127.13.100.2:36851
I20260929 18:41:04.544502 30537 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.100.2:36851 every 8 connection(s)
I20260929 18:41:04.545341 30406 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/ts-1/data/info.pb
I20260929 18:41:04.547742 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 30406
I20260929 18:41:04.548195 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.100.3:46121
--local_ip_for_outbound_sockets=127.13.100.3
--tserver_master_addrs=127.13.100.62:34215
--webserver_port=34335
--webserver_interface=127.13.100.3
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.3.keytab
--principal=kudu/127.13.100.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.100.20:44697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 18:41:04.548900 30540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:04.547474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:34215 (local address 127.13.100.2:58199)
0929 18:41:04.547977 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:04.548113 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.62:34215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":153,"threads_started":1}
W20260929 18:41:04.549409 30538 heartbeater.cc:644] Failed to heartbeat to 127.13.100.62:34215 (0 consecutive failures): Network error: Failed to ping master at 127.13.100.62:34215: Client connection negotiation failed: client connection to 127.13.100.62:34215: connect: Connection refused (error 111)
I20260929 18:41:04.550189 30540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:04.549918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:34215 (local address 127.13.100.2:35341)
0929 18:41:04.550065 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:04.550134 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.62:34215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260929 18:41:04.550809 30540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:04.550582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:34215 (local address 127.13.100.2:45269)
0929 18:41:04.550706 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:04.550752 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.62:34215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260929 18:41:04.550983 30538 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 18:41:04.659947 30541 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:41:04.660212 30541 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:41:04.660259 30541 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:41:04.664276 30541 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:41:04.664379 30541 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:41:04.664451 30541 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.100.3
I20260929 18:41:04.669636 30541 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.100.20:44697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.3.keytab
--principal=kudu/127.13.100.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.100.3:46121
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.100.3
--webserver_port=34335
--tserver_master_addrs=127.13.100.62:34215
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30541
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.100.3
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:41:04.670750 30541 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:41:04.671720 30541 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:41:04.681923 30546 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:41:04.681924 30547 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:41:04.681974 30541 server_base.cc:1034] running on GCE node
W20260929 18:41:04.681998 30549 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:41:04.682541 30541 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:41:04.683199 30541 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:41:04.684405 30541 hybrid_clock.cc:648] HybridClock initialized: now 1790707264684368 us; error 58 us; skew 500 ppm
Sep 29 18:41:04 dist-test-slave-j1xs krb5kdc[29084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707264, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:41:04.687224 30541 init.cc:377] Logged in from keytab as kudu/127.13.100.3@KRBTEST.COM (short username kudu)
I20260929 18:41:04.688037 30541 webserver.cc:460] Webserver started at http://127.13.100.3:34335/ using document root <none> and password file <none>
I20260929 18:41:04.688556 30541 fs_manager.cc:359] Metadata directory not provided
I20260929 18:41:04.688623 30541 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:41:04.692377 30541 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260929 18:41:04.694552 30556 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:41:04.695819 30541 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260929 18:41:04.695956 30541 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/ts-2/wal
uuid: "0f8af5f5a4024455869ea7a896d2364f"
format_stamp: "Formatted at 2026-09-29 18:40:22 on dist-test-slave-j1xs"
server_key: "522c02267cf80687cb19146bab7805ee"
server_key_iv: "b781fd704777451fefe12e45a4893748"
server_key_version: "encryptionkey@0"
I20260929 18:41:04.696519 30541 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:41:04.718281 30541 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:41:04.721675 30541 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:41:04.721910 30541 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:41:04.722533 30541 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:41:04.723342 30541 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:41:04.723389 30541 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:41:04.723495 30541 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:41:04.723524 30541 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:41:04.732322 30541 rpc_server.cc:307] RPC server started. Bound to: 127.13.100.3:46121
I20260929 18:41:04.732348 30669 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.100.3:46121 every 8 connection(s)
I20260929 18:41:04.733290 30541 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707036041078-13712-0/minicluster-data/ts-2/data/info.pb
I20260929 18:41:04.735199 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 30541
I20260929 18:41:04.736941 30672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:04.735435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:34215 (local address 127.13.100.3:57525)
0929 18:41:04.736148 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:04.736270 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.62:34215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":145,"threads_started":1}
W20260929 18:41:04.737345 30670 heartbeater.cc:644] Failed to heartbeat to 127.13.100.62:34215 (0 consecutive failures): Network error: Failed to ping master at 127.13.100.62:34215: Client connection negotiation failed: client connection to 127.13.100.62:34215: connect: Connection refused (error 111)
I20260929 18:41:04.737972 30672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:04.737707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:34215 (local address 127.13.100.3:57785)
0929 18:41:04.737821 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:04.737872 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.62:34215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260929 18:41:04.738566 30672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:04.738348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:34215 (local address 127.13.100.3:39855)
0929 18:41:04.738453 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:04.738496 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.62:34215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260929 18:41:04.738792 30670 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 18:41:05.423727 30674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:05.423249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:34215 (local address 127.13.100.1:33299)
0929 18:41:05.423535 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:05.423597 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.62:34215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":77,"threads_started":1}
I20260929 18:41:05.551966 30675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:05.551569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:34215 (local address 127.13.100.2:40101)
0929 18:41:05.551818 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:05.551882 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.62:34215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":74,"threads_started":1}
I20260929 18:41:05.739780 30676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:05.739363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:34215 (local address 127.13.100.3:58143)
0929 18:41:05.739637 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:05.739690 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.62:34215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":75,"threads_started":1}
I20260929 18:41:05.887171 30677 raft_consensus.cc:482] T 9b9dac414f55412ba253d2c11113efee P dfa07a7f1fec49beb458bd72df7589e7 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:41:05.887393 30677 raft_consensus.cc:504] T 9b9dac414f55412ba253d2c11113efee P dfa07a7f1fec49beb458bd72df7589e7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f8af5f5a4024455869ea7a896d2364f" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 46121 } } peers { permanent_uuid: "248d7dfbf266486b99f1abfc09afcb8e" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 36851 } } peers { permanent_uuid: "dfa07a7f1fec49beb458bd72df7589e7" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 36845 } }
I20260929 18:41:05.888540 30677 leader_election.cc:288] T 9b9dac414f55412ba253d2c11113efee P dfa07a7f1fec49beb458bd72df7589e7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0f8af5f5a4024455869ea7a896d2364f (127.13.100.3:46121), 248d7dfbf266486b99f1abfc09afcb8e (127.13.100.2:36851)
Sep 29 18:41:05 dist-test-slave-j1xs krb5kdc[29084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707264, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.1@KRBTEST.COM for kudu/127.13.100.3@KRBTEST.COM
Sep 29 18:41:05 dist-test-slave-j1xs krb5kdc[29084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707264, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.1@KRBTEST.COM for kudu/127.13.100.2@KRBTEST.COM
I20260929 18:41:05.907438 30680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:05.889220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.1:57071 (local address 127.13.100.2:36851)
0929 18:41:05.889570 (+ 350us) server_negotiation.cc:206] Beginning negotiation
0929 18:41:05.889582 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 18:41:05.889617 (+ 35us) server_negotiation.cc:407] Connection header received
0929 18:41:05.890444 (+ 827us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:41:05.890452 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:41:05.890761 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:41:05.891085 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:41:05.892086 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:05.893023 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:05.894415 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:05.894891 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:05.895065 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:05.902683 (+ 7618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:41:05.902687 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:41:05.902698 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:41:05.902726 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:41:05.904754 (+ 2028us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:41:05.905340 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:41:05.905344 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:41:05.905348 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:41:05.905579 (+ 231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:41:05.905927 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:41:05.905934 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:41:05.905936 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:41:05.906392 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:41:05.906560 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0929 18:41:05.906984 (+ 424us) server_negotiation.cc:299] Negotiation successful
0929 18:41:05.907233 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":64,"threads_started":1}
I20260929 18:41:05.907482 30678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:05.888933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.1:57807 (local address 127.13.100.3:46121)
0929 18:41:05.889271 (+ 338us) server_negotiation.cc:206] Beginning negotiation
0929 18:41:05.889299 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0929 18:41:05.889345 (+ 46us) server_negotiation.cc:407] Connection header received
0929 18:41:05.890308 (+ 963us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:41:05.890317 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:41:05.890762 (+ 445us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:41:05.891085 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:41:05.892078 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:05.893039 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:05.894417 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:05.894875 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:05.895056 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:05.902300 (+ 7244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:41:05.902306 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:41:05.902319 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:41:05.902348 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:41:05.904352 (+ 2004us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:41:05.905156 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:41:05.905165 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:41:05.905171 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:41:05.905322 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:41:05.905694 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:41:05.905697 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:41:05.905699 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:41:05.906271 (+ 572us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:41:05.906374 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 18:41:05.906967 (+ 593us) server_negotiation.cc:299] Negotiation successful
0929 18:41:05.907277 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":222,"thread_start_us":117,"threads_started":1}
I20260929 18:41:05.907477 30674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:05.888709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.3:46121 (local address 127.13.100.1:57807)
0929 18:41:05.888853 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:05.888877 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 18:41:05.889006 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:41:05.891116 (+ 2110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:41:05.891121 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:41:05.891469 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:41:05.891903 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:41:05.891916 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:41:05.893179 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:41:05.893190 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:41:05.894298 (+ 1108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:41:05.894307 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:41:05.894999 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:41:05.895003 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:41:05.895260 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:05.900348 (+ 5088us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:41:05.900366 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:41:05.902046 (+ 1680us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:41:05.904586 (+ 2540us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:41:05.904597 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:41:05.904613 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:41:05.905035 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:41:05.905422 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:41:05.905427 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:41:05.905428 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:41:05.905594 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:41:05.906386 (+ 792us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:41:05.906393 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:41:05.906670 (+ 277us) client_negotiation.cc:769] Sending connection context
0929 18:41:05.906856 (+ 186us) client_negotiation.cc:240] Negotiation successful
0929 18:41:05.907212 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48,"mutex_wait_us":182,"spinlock_wait_cycles":66944}
I20260929 18:41:05.907490 30679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:05.889001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.2:36851 (local address 127.13.100.1:57071)
0929 18:41:05.889309 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:05.889320 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 18:41:05.889398 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:41:05.891546 (+ 2148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:41:05.891548 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:41:05.891710 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:41:05.891903 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:41:05.891916 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:41:05.893189 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:41:05.893192 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:41:05.894294 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:41:05.894302 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:41:05.895012 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:41:05.895016 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:41:05.895264 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:05.900344 (+ 5080us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:41:05.900367 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:41:05.902417 (+ 2050us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:41:05.904966 (+ 2549us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:41:05.904969 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:41:05.904972 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:41:05.905233 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:41:05.905690 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:41:05.905693 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:41:05.905695 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:41:05.905798 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:41:05.906504 (+ 706us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:41:05.906506 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:41:05.906663 (+ 157us) client_negotiation.cc:769] Sending connection context
0929 18:41:05.906999 (+ 336us) client_negotiation.cc:240] Negotiation successful
0929 18:41:05.907296 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"mutex_wait_us":195,"spinlock_wait_cycles":8960,"thread_start_us":60,"threads_started":1}
W20260929 18:41:05.910490 30289 leader_election.cc:341] T 9b9dac414f55412ba253d2c11113efee P dfa07a7f1fec49beb458bd72df7589e7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 248d7dfbf266486b99f1abfc09afcb8e (127.13.100.2:36851): Not found: Tablet not found: 9b9dac414f55412ba253d2c11113efee
W20260929 18:41:05.910742 30289 leader_election.cc:341] T 9b9dac414f55412ba253d2c11113efee P dfa07a7f1fec49beb458bd72df7589e7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0f8af5f5a4024455869ea7a896d2364f (127.13.100.3:46121): Not found: Tablet not found: 9b9dac414f55412ba253d2c11113efee
I20260929 18:41:05.910818 30289 leader_election.cc:302] T 9b9dac414f55412ba253d2c11113efee P dfa07a7f1fec49beb458bd72df7589e7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dfa07a7f1fec49beb458bd72df7589e7; no voters: 0f8af5f5a4024455869ea7a896d2364f, 248d7dfbf266486b99f1abfc09afcb8e
I20260929 18:41:05.911129 30677 raft_consensus.cc:2726] T 9b9dac414f55412ba253d2c11113efee P dfa07a7f1fec49beb458bd72df7589e7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 18:41:06.424954 30681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:06.424512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:34215 (local address 127.13.100.1:39387)
0929 18:41:06.424792 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:06.424859 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.62:34215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":89,"threads_started":1}
I20260929 18:41:06.553119 30682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:06.552718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:34215 (local address 127.13.100.2:47939)
0929 18:41:06.552946 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:06.553012 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.62:34215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":73,"threads_started":1}
I20260929 18:41:06.740952 30683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:06.740496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:34215 (local address 127.13.100.3:37127)
0929 18:41:06.740807 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:06.740868 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.62:34215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":71,"threads_started":1}
I20260929 18:41:07.426200 30684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:07.425721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:34215 (local address 127.13.100.1:56299)
0929 18:41:07.426047 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:07.426102 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.62:34215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":72,"threads_started":1}
I20260929 18:41:07.554353 30685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:07.553934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:34215 (local address 127.13.100.2:40731)
0929 18:41:07.554206 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:07.554260 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.62:34215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":81,"threads_started":1}
I20260929 18:41:07.648634 30686 raft_consensus.cc:482] T 9b9dac414f55412ba253d2c11113efee P dfa07a7f1fec49beb458bd72df7589e7 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:41:07.648814 30686 raft_consensus.cc:504] T 9b9dac414f55412ba253d2c11113efee P dfa07a7f1fec49beb458bd72df7589e7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f8af5f5a4024455869ea7a896d2364f" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 46121 } } peers { permanent_uuid: "248d7dfbf266486b99f1abfc09afcb8e" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 36851 } } peers { permanent_uuid: "dfa07a7f1fec49beb458bd72df7589e7" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 36845 } }
I20260929 18:41:07.649200 30686 leader_election.cc:288] T 9b9dac414f55412ba253d2c11113efee P dfa07a7f1fec49beb458bd72df7589e7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0f8af5f5a4024455869ea7a896d2364f (127.13.100.3:46121), 248d7dfbf266486b99f1abfc09afcb8e (127.13.100.2:36851)
W20260929 18:41:07.650249 30289 leader_election.cc:341] T 9b9dac414f55412ba253d2c11113efee P dfa07a7f1fec49beb458bd72df7589e7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0f8af5f5a4024455869ea7a896d2364f (127.13.100.3:46121): Not found: Tablet not found: 9b9dac414f55412ba253d2c11113efee
W20260929 18:41:07.650425 30289 leader_election.cc:341] T 9b9dac414f55412ba253d2c11113efee P dfa07a7f1fec49beb458bd72df7589e7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 248d7dfbf266486b99f1abfc09afcb8e (127.13.100.2:36851): Not found: Tablet not found: 9b9dac414f55412ba253d2c11113efee
I20260929 18:41:07.650512 30289 leader_election.cc:302] T 9b9dac414f55412ba253d2c11113efee P dfa07a7f1fec49beb458bd72df7589e7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dfa07a7f1fec49beb458bd72df7589e7; no voters: 0f8af5f5a4024455869ea7a896d2364f, 248d7dfbf266486b99f1abfc09afcb8e
I20260929 18:41:07.650736 30686 raft_consensus.cc:2726] T 9b9dac414f55412ba253d2c11113efee P dfa07a7f1fec49beb458bd72df7589e7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 18:41:07.742214 30687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:07.741737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:34215 (local address 127.13.100.3:36731)
0929 18:41:07.742053 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:07.742111 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.62:34215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":70,"threads_started":1}
I20260929 18:41:08.427444 30688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:08.427016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:34215 (local address 127.13.100.1:52345)
0929 18:41:08.427285 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:08.427350 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.62:34215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":76,"threads_started":1}
I20260929 18:41:08.555617 30689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:08.555153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:34215 (local address 127.13.100.2:35713)
0929 18:41:08.555461 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:08.555517 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.62:34215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":104,"threads_started":1}
I20260929 18:41:08.743357 30690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:08.742958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:34215 (local address 127.13.100.3:39563)
0929 18:41:08.743218 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:08.743271 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.62:34215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":79,"threads_started":1}
I20260929 18:41:09.428903 30691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:09.428401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:34215 (local address 127.13.100.1:55765)
0929 18:41:09.428739 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:09.428802 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.62:34215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":86,"threads_started":1}
I20260929 18:41:09.557034 30692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:09.556482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:34215 (local address 127.13.100.2:40895)
0929 18:41:09.556864 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:09.556917 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.62:34215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":83,"threads_started":1}
I20260929 18:41:09.744477 30693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:09.744072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:34215 (local address 127.13.100.3:49453)
0929 18:41:09.744334 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:09.744386 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.62:34215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":87,"threads_started":1}
I20260929 18:41:10.430253 30694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:10.429801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:34215 (local address 127.13.100.1:39397)
0929 18:41:10.430100 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:10.430157 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.62:34215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":129,"threads_started":1}
I20260929 18:41:10.558339 30695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:10.557871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:34215 (local address 127.13.100.2:46345)
0929 18:41:10.558187 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:10.558246 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.62:34215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":71,"threads_started":1}
I20260929 18:41:10.725708 30696 raft_consensus.cc:482] T 9b9dac414f55412ba253d2c11113efee P dfa07a7f1fec49beb458bd72df7589e7 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:41:10.725867 30696 raft_consensus.cc:504] T 9b9dac414f55412ba253d2c11113efee P dfa07a7f1fec49beb458bd72df7589e7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f8af5f5a4024455869ea7a896d2364f" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 46121 } } peers { permanent_uuid: "248d7dfbf266486b99f1abfc09afcb8e" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 36851 } } peers { permanent_uuid: "dfa07a7f1fec49beb458bd72df7589e7" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 36845 } }
I20260929 18:41:10.726328 30696 leader_election.cc:288] T 9b9dac414f55412ba253d2c11113efee P dfa07a7f1fec49beb458bd72df7589e7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0f8af5f5a4024455869ea7a896d2364f (127.13.100.3:46121), 248d7dfbf266486b99f1abfc09afcb8e (127.13.100.2:36851)
W20260929 18:41:10.727167 30289 leader_election.cc:341] T 9b9dac414f55412ba253d2c11113efee P dfa07a7f1fec49beb458bd72df7589e7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0f8af5f5a4024455869ea7a896d2364f (127.13.100.3:46121): Not found: Tablet not found: 9b9dac414f55412ba253d2c11113efee
W20260929 18:41:10.727336 30289 leader_election.cc:341] T 9b9dac414f55412ba253d2c11113efee P dfa07a7f1fec49beb458bd72df7589e7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 248d7dfbf266486b99f1abfc09afcb8e (127.13.100.2:36851): Not found: Tablet not found: 9b9dac414f55412ba253d2c11113efee
I20260929 18:41:10.727402 30289 leader_election.cc:302] T 9b9dac414f55412ba253d2c11113efee P dfa07a7f1fec49beb458bd72df7589e7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dfa07a7f1fec49beb458bd72df7589e7; no voters: 0f8af5f5a4024455869ea7a896d2364f, 248d7dfbf266486b99f1abfc09afcb8e
I20260929 18:41:10.727608 30696 raft_consensus.cc:2726] T 9b9dac414f55412ba253d2c11113efee P dfa07a7f1fec49beb458bd72df7589e7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 18:41:10.745560 30697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:10.745195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:34215 (local address 127.13.100.3:57099)
0929 18:41:10.745432 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:10.745484 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.62:34215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":68,"threads_started":1}
I20260929 18:41:11.431401 30698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:11.430968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:34215 (local address 127.13.100.1:44131)
0929 18:41:11.431240 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:11.431303 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.62:34215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":71,"threads_started":1}
I20260929 18:41:11.559473 30699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:11.559052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:34215 (local address 127.13.100.2:35435)
0929 18:41:11.559320 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:11.559374 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.62:34215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":70,"threads_started":1}
I20260929 18:41:11.746768 30700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:11.746297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:34215 (local address 127.13.100.3:36545)
0929 18:41:11.746615 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:11.746677 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.62:34215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":83,"threads_started":1}
I20260929 18:41:12.432664 30701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:12.432190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:34215 (local address 127.13.100.1:47497)
0929 18:41:12.432506 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:12.432571 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.62:34215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":87,"threads_started":1}
I20260929 18:41:12.560746 30702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:12.560300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:34215 (local address 127.13.100.2:57169)
0929 18:41:12.560574 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:12.560651 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.62:34215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":84,"threads_started":1}
I20260929 18:41:12.747961 30703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:12.747580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:34215 (local address 127.13.100.3:37751)
0929 18:41:12.747817 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:12.747873 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.62:34215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":83,"threads_started":1}
I20260929 18:41:13.434129 30704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:13.433561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:34215 (local address 127.13.100.1:41083)
0929 18:41:13.433900 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:13.433993 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.62:34215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":95,"threads_started":1}
I20260929 18:41:13.562167 30705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:13.561582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:34215 (local address 127.13.100.2:39673)
0929 18:41:13.561877 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:13.562070 (+ 193us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.62:34215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":89,"threads_started":1}
I20260929 18:41:13.749127 30706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:13.748730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:34215 (local address 127.13.100.3:59905)
0929 18:41:13.748965 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:13.749020 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.62:34215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":74,"threads_started":1}
I20260929 18:41:14.435585 30707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:14.435005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:34215 (local address 127.13.100.1:55195)
0929 18:41:14.435419 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:14.435487 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.62:34215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":81,"threads_started":1}
I20260929 18:41:14.449667 30708 raft_consensus.cc:482] T 9b9dac414f55412ba253d2c11113efee P dfa07a7f1fec49beb458bd72df7589e7 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:41:14.449818 30708 raft_consensus.cc:504] T 9b9dac414f55412ba253d2c11113efee P dfa07a7f1fec49beb458bd72df7589e7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f8af5f5a4024455869ea7a896d2364f" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 46121 } } peers { permanent_uuid: "248d7dfbf266486b99f1abfc09afcb8e" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 36851 } } peers { permanent_uuid: "dfa07a7f1fec49beb458bd72df7589e7" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 36845 } }
I20260929 18:41:14.450300 30708 leader_election.cc:288] T 9b9dac414f55412ba253d2c11113efee P dfa07a7f1fec49beb458bd72df7589e7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0f8af5f5a4024455869ea7a896d2364f (127.13.100.3:46121), 248d7dfbf266486b99f1abfc09afcb8e (127.13.100.2:36851)
W20260929 18:41:14.451018 30289 leader_election.cc:341] T 9b9dac414f55412ba253d2c11113efee P dfa07a7f1fec49beb458bd72df7589e7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0f8af5f5a4024455869ea7a896d2364f (127.13.100.3:46121): Not found: Tablet not found: 9b9dac414f55412ba253d2c11113efee
W20260929 18:41:14.451161 30289 leader_election.cc:341] T 9b9dac414f55412ba253d2c11113efee P dfa07a7f1fec49beb458bd72df7589e7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 248d7dfbf266486b99f1abfc09afcb8e (127.13.100.2:36851): Not found: Tablet not found: 9b9dac414f55412ba253d2c11113efee
I20260929 18:41:14.451238 30289 leader_election.cc:302] T 9b9dac414f55412ba253d2c11113efee P dfa07a7f1fec49beb458bd72df7589e7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dfa07a7f1fec49beb458bd72df7589e7; no voters: 0f8af5f5a4024455869ea7a896d2364f, 248d7dfbf266486b99f1abfc09afcb8e
I20260929 18:41:14.451366 30708 raft_consensus.cc:2726] T 9b9dac414f55412ba253d2c11113efee P dfa07a7f1fec49beb458bd72df7589e7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 18:41:14.563360 30709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:14.562938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:34215 (local address 127.13.100.2:50093)
0929 18:41:14.563201 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:14.563268 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.62:34215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":74,"threads_started":1}
I20260929 18:41:14.750483 30710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:14.750065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:34215 (local address 127.13.100.3:50797)
0929 18:41:14.750332 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:14.750392 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.62:34215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":85,"threads_started":1}
I20260929 18:41:15.437029 30711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:15.436542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:34215 (local address 127.13.100.1:54373)
0929 18:41:15.436831 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:15.436896 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.62:34215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":87,"threads_started":1}
I20260929 18:41:15.564693 30712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:15.564215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:34215 (local address 127.13.100.2:39059)
0929 18:41:15.564525 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:15.564585 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.62:34215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":85,"threads_started":1}
I20260929 18:41:15.751601 30713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:15.751188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:34215 (local address 127.13.100.3:36945)
0929 18:41:15.751433 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:15.751507 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.62:34215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":70,"threads_started":1}
I20260929 18:41:16.438354 30714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:16.437833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:34215 (local address 127.13.100.1:39413)
0929 18:41:16.438194 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:16.438255 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.62:34215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":152,"threads_started":1}
I20260929 18:41:16.566155 30715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:16.565606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:34215 (local address 127.13.100.2:46201)
0929 18:41:16.565965 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:16.566026 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.62:34215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":94,"threads_started":1}
I20260929 18:41:16.752771 30716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:16.752332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:34215 (local address 127.13.100.3:47115)
0929 18:41:16.752606 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:16.752671 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.62:34215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":73,"threads_started":1}
I20260929 18:41:17.439492 30717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:17.439109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:34215 (local address 127.13.100.1:33611)
0929 18:41:17.439346 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:17.439400 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.62:34215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":72,"threads_started":1}
I20260929 18:41:17.567279 30718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:17.566896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:34215 (local address 127.13.100.2:38553)
0929 18:41:17.567132 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:17.567186 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.62:34215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":74,"threads_started":1}
I20260929 18:41:17.753969 30719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:17.753508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:34215 (local address 127.13.100.3:51935)
0929 18:41:17.753757 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:17.753813 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.62:34215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":72,"threads_started":1}
I20260929 18:41:18.440770 30720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:18.440282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:34215 (local address 127.13.100.1:50779)
0929 18:41:18.440610 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:18.440670 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.62:34215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":106,"threads_started":1}
I20260929 18:41:18.568535 30721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:18.568092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:34215 (local address 127.13.100.2:46937)
0929 18:41:18.568368 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:18.568434 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.62:34215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":79,"threads_started":1}
I20260929 18:41:18.755127 30722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:18.754694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:34215 (local address 127.13.100.3:34655)
0929 18:41:18.754978 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:18.755036 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.62:34215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":79,"threads_started":1}
I20260929 18:41:19.106323 30723 raft_consensus.cc:482] T 9b9dac414f55412ba253d2c11113efee P dfa07a7f1fec49beb458bd72df7589e7 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:41:19.106451 30723 raft_consensus.cc:504] T 9b9dac414f55412ba253d2c11113efee P dfa07a7f1fec49beb458bd72df7589e7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f8af5f5a4024455869ea7a896d2364f" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 46121 } } peers { permanent_uuid: "248d7dfbf266486b99f1abfc09afcb8e" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 36851 } } peers { permanent_uuid: "dfa07a7f1fec49beb458bd72df7589e7" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 36845 } }
I20260929 18:41:19.106880 30723 leader_election.cc:288] T 9b9dac414f55412ba253d2c11113efee P dfa07a7f1fec49beb458bd72df7589e7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0f8af5f5a4024455869ea7a896d2364f (127.13.100.3:46121), 248d7dfbf266486b99f1abfc09afcb8e (127.13.100.2:36851)
W20260929 18:41:19.107762 30289 leader_election.cc:341] T 9b9dac414f55412ba253d2c11113efee P dfa07a7f1fec49beb458bd72df7589e7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0f8af5f5a4024455869ea7a896d2364f (127.13.100.3:46121): Not found: Tablet not found: 9b9dac414f55412ba253d2c11113efee
W20260929 18:41:19.107955 30289 leader_election.cc:341] T 9b9dac414f55412ba253d2c11113efee P dfa07a7f1fec49beb458bd72df7589e7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 248d7dfbf266486b99f1abfc09afcb8e (127.13.100.2:36851): Not found: Tablet not found: 9b9dac414f55412ba253d2c11113efee
I20260929 18:41:19.108035 30289 leader_election.cc:302] T 9b9dac414f55412ba253d2c11113efee P dfa07a7f1fec49beb458bd72df7589e7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dfa07a7f1fec49beb458bd72df7589e7; no voters: 0f8af5f5a4024455869ea7a896d2364f, 248d7dfbf266486b99f1abfc09afcb8e
I20260929 18:41:19.108242 30723 raft_consensus.cc:2726] T 9b9dac414f55412ba253d2c11113efee P dfa07a7f1fec49beb458bd72df7589e7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 18:41:19.442320 30724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:19.441813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:34215 (local address 127.13.100.1:36215)
0929 18:41:19.442121 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:19.442181 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.62:34215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":142,"threads_started":1}
I20260929 18:41:19.569948 30725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:19.569449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:34215 (local address 127.13.100.2:58965)
0929 18:41:19.569722 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:19.569778 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.62:34215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":82,"threads_started":1}
I20260929 18:41:19.737627 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 30233
I20260929 18:41:19.743528 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 30406
I20260929 18:41:19.749168 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 30541
2026-09-29T18:41:19Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59420 ms)
[----------] 30 tests from SecurityITest (243703 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707036041078-13712-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:41:19 dist-test-slave-j1xs krb5kdc[30728](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:41:19 dist-test-slave-j1xs krb5kdc[30728](info): set up 2 sockets
Sep 29 18:41:19 dist-test-slave-j1xs krb5kdc[30728](info): commencing operation
krb5kdc: starting...
W20260929 18:41:20.809715 13712 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707036041078-13712-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707036041078-13712-0/krb5kdc/test-user.keytab.
Sep 29 18:41:20 dist-test-slave-j1xs krb5kdc[30728](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707280, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:41:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:41:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.100.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.100.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.100.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.62.keytab.
Entry for principal kudu/127.13.100.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.62.keytab.
I20260929 18:41:20.931126 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707036041078-13712-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707036041078-13712-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707036041078-13712-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.100.62:40053
--webserver_interface=127.13.100.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.62.keytab
--principal=kudu/127.13.100.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.100.20:32911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.100.62:40053
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707036041078-13712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707036041078-13712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707036041078-13712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:41:21.039466 30741 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:41:21.039757 30741 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:41:21.039829 30741 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:41:21.043761 30741 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:41:21.043853 30741 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:41:21.043898 30741 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:41:21.043959 30741 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:41:21.043998 30741 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:41:21.049295 30741 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.100.20:32911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707036041078-13712-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707036041078-13712-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.100.62:40053
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.62.keytab
--principal=kudu/127.13.100.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.100.62:40053
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.100.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30741
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707036041078-13712-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:41:21.050356 30741 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:41:21.051326 30741 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:41:21.059742 30747 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:41:21.059767 30746 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:41:21.059767 30749 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:41:21.059873 30741 server_base.cc:1034] running on GCE node
I20260929 18:41:21.060573 30741 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:41:21.061676 30741 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:41:21.062882 30741 hybrid_clock.cc:648] HybridClock initialized: now 1790707281062857 us; error 29 us; skew 500 ppm
Sep 29 18:41:21 dist-test-slave-j1xs krb5kdc[30728](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707281, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:41:21.065564 30741 init.cc:377] Logged in from keytab as kudu/127.13.100.62@KRBTEST.COM (short username kudu)
I20260929 18:41:21.066457 30741 webserver.cc:460] Webserver started at http://127.13.100.62:35773/ using document root <none> and password file <none>
I20260929 18:41:21.066998 30741 fs_manager.cc:359] Metadata directory not provided
I20260929 18:41:21.067090 30741 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:41:21.067312 30741 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:41:21.069249 30741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707036041078-13712-0/minicluster-data/master-0/data/instance:
uuid: "3485b2fdf4c340f0b73d535354170a23"
format_stamp: "Formatted at 2026-09-29 18:41:21 on dist-test-slave-j1xs"
server_key: "2cc0aba63d60f5827ea5ef802e10a25e"
server_key_iv: "9fd5512d8d6525f410a6735b796f4e29"
server_key_version: "encryptionkey@0"
I20260929 18:41:21.069813 30741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707036041078-13712-0/minicluster-data/master-0/wal/instance:
uuid: "3485b2fdf4c340f0b73d535354170a23"
format_stamp: "Formatted at 2026-09-29 18:41:21 on dist-test-slave-j1xs"
server_key: "2cc0aba63d60f5827ea5ef802e10a25e"
server_key_iv: "9fd5512d8d6525f410a6735b796f4e29"
server_key_version: "encryptionkey@0"
I20260929 18:41:21.073689 30741 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260929 18:41:21.076071 30756 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:41:21.077340 30741 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:41:21.077478 30741 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707036041078-13712-0/minicluster-data/master-0/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707036041078-13712-0/minicluster-data/master-0/wal
uuid: "3485b2fdf4c340f0b73d535354170a23"
format_stamp: "Formatted at 2026-09-29 18:41:21 on dist-test-slave-j1xs"
server_key: "2cc0aba63d60f5827ea5ef802e10a25e"
server_key_iv: "9fd5512d8d6525f410a6735b796f4e29"
server_key_version: "encryptionkey@0"
I20260929 18:41:21.077587 30741 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707036041078-13712-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707036041078-13712-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707036041078-13712-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:41:21.092708 30741 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:41:21.095520 30741 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:41:21.095715 30741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:41:21.102922 30741 rpc_server.cc:307] RPC server started. Bound to: 127.13.100.62:40053
I20260929 18:41:21.102974 30808 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.100.62:40053 every 8 connection(s)
I20260929 18:41:21.103897 30741 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
I20260929 18:41:21.106652 30809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:41:21.107216 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 30741
I20260929 18:41:21.107402 13712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707036041078-13712-0/minicluster-data/master-0/wal/instance
I20260929 18:41:21.107724 13712 external_mini_cluster.cc:1428] Setting key 06ea818c174adfa8548fc5aa043a8874
I20260929 18:41:21.113322 30809 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3485b2fdf4c340f0b73d535354170a23: Bootstrap starting.
I20260929 18:41:21.115969 30809 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3485b2fdf4c340f0b73d535354170a23: Neither blocks nor log segments found. Creating new log.
I20260929 18:41:21.116684 30809 log.cc:824] T 00000000000000000000000000000000 P 3485b2fdf4c340f0b73d535354170a23: Log is configured to *not* fsync() on all Append() calls
Sep 29 18:41:21 dist-test-slave-j1xs krb5kdc[30728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707280, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.100.62@KRBTEST.COM
I20260929 18:41:21.118688 30809 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3485b2fdf4c340f0b73d535354170a23: No bootstrap required, opened a new log
I20260929 18:41:21.122568 30812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:21.109005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38096 (local address 127.13.100.62:40053)
0929 18:41:21.109422 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0929 18:41:21.109441 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 18:41:21.109558 (+ 117us) server_negotiation.cc:407] Connection header received
0929 18:41:21.110439 (+ 881us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:41:21.110462 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:41:21.110857 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:41:21.111225 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:41:21.111969 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:21.113434 (+ 1465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:21.114403 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:21.115089 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:21.115249 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:21.117976 (+ 2727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:41:21.117981 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:41:21.117993 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:41:21.118018 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:41:21.119805 (+ 1787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:41:21.120299 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:41:21.120303 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:41:21.120308 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:41:21.120409 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:41:21.120714 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:41:21.120716 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:41:21.120717 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:41:21.121079 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:41:21.121225 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0929 18:41:21.121498 (+ 273us) server_negotiation.cc:299] Negotiation successful
0929 18:41:21.121784 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":107,"threads_started":1}
I20260929 18:41:21.122550 30809 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3485b2fdf4c340f0b73d535354170a23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3485b2fdf4c340f0b73d535354170a23" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 40053 } }
I20260929 18:41:21.122809 30809 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3485b2fdf4c340f0b73d535354170a23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:41:21.122864 30809 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3485b2fdf4c340f0b73d535354170a23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3485b2fdf4c340f0b73d535354170a23, State: Initialized, Role: FOLLOWER
I20260929 18:41:21.123251 30809 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3485b2fdf4c340f0b73d535354170a23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3485b2fdf4c340f0b73d535354170a23" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 40053 } }
I20260929 18:41:21.123374 30809 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3485b2fdf4c340f0b73d535354170a23 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:41:21.123452 30809 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3485b2fdf4c340f0b73d535354170a23 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:41:21.123543 30809 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3485b2fdf4c340f0b73d535354170a23 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:41:21.124354 30809 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3485b2fdf4c340f0b73d535354170a23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3485b2fdf4c340f0b73d535354170a23" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 40053 } }
I20260929 18:41:21.124727 30809 leader_election.cc:302] T 00000000000000000000000000000000 P 3485b2fdf4c340f0b73d535354170a23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3485b2fdf4c340f0b73d535354170a23; no voters:
I20260929 18:41:21.125052 30809 leader_election.cc:288] T 00000000000000000000000000000000 P 3485b2fdf4c340f0b73d535354170a23 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:41:21.125175 30814 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3485b2fdf4c340f0b73d535354170a23 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:41:21.125393 30814 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3485b2fdf4c340f0b73d535354170a23 [term 1 LEADER]: Becoming Leader. State: Replica: 3485b2fdf4c340f0b73d535354170a23, State: Running, Role: LEADER
I20260929 18:41:21.125741 30814 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3485b2fdf4c340f0b73d535354170a23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3485b2fdf4c340f0b73d535354170a23" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 40053 } }
I20260929 18:41:21.126259 30809 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3485b2fdf4c340f0b73d535354170a23 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:41:21.127460 30816 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3485b2fdf4c340f0b73d535354170a23 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3485b2fdf4c340f0b73d535354170a23. Latest consensus state: current_term: 1 leader_uuid: "3485b2fdf4c340f0b73d535354170a23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3485b2fdf4c340f0b73d535354170a23" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 40053 } } }
I20260929 18:41:21.127486 30815 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3485b2fdf4c340f0b73d535354170a23 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3485b2fdf4c340f0b73d535354170a23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3485b2fdf4c340f0b73d535354170a23" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 40053 } } }
I20260929 18:41:21.127605 30816 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3485b2fdf4c340f0b73d535354170a23 [sys.catalog]: This master's current role is: LEADER
I20260929 18:41:21.127661 30815 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3485b2fdf4c340f0b73d535354170a23 [sys.catalog]: This master's current role is: LEADER
W20260929 18:41:21.131610 30829 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3485b2fdf4c340f0b73d535354170a23: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:41:21.131688 30829 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:41:21.131742 30823 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:41:21.132553 30823 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:41:21.138073 30823 catalog_manager.cc:1347] Generated new cluster ID: 7ea7690273c6413484166d8df1a3aa31
I20260929 18:41:21.138140 30823 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:41:21.150056 30823 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:41:21.151038 30823 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:41:21.160302 30823 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3485b2fdf4c340f0b73d535354170a23: Generated new TSK 0
I20260929 18:41:21.160920 30823 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.100.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.100.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.100.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.1.keytab.
Entry for principal kudu/127.13.100.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.1.keytab.
I20260929 18:41:21.193048 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707036041078-13712-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707036041078-13712-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707036041078-13712-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707036041078-13712-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.100.1:0
--local_ip_for_outbound_sockets=127.13.100.1
--webserver_interface=127.13.100.1
--webserver_port=0
--tserver_master_addrs=127.13.100.62:40053
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.1.keytab
--principal=kudu/127.13.100.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.100.20:32911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707036041078-13712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707036041078-13712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707036041078-13712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:41:21.304247 30835 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:41:21.304612 30835 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:41:21.304661 30835 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:41:21.308808 30835 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:41:21.308868 30835 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:41:21.308928 30835 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.100.1
I20260929 18:41:21.314222 30835 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.100.20:32911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707036041078-13712-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707036041078-13712-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.1.keytab
--principal=kudu/127.13.100.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.100.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707036041078-13712-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.100.1
--webserver_port=0
--tserver_master_addrs=127.13.100.62:40053
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30835
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.100.1
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707036041078-13712-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:41:21.315292 30835 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:41:21.316339 30835 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:41:21.326185 30843 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:41:21.326205 30840 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:41:21.326403 30835 server_base.cc:1034] running on GCE node
W20260929 18:41:21.326205 30841 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:41:21.326961 30835 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:41:21.327718 30835 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:41:21.328902 30835 hybrid_clock.cc:648] HybridClock initialized: now 1790707281328889 us; error 32 us; skew 500 ppm
Sep 29 18:41:21 dist-test-slave-j1xs krb5kdc[30728](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707281, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:41:21.331638 30835 init.cc:377] Logged in from keytab as kudu/127.13.100.1@KRBTEST.COM (short username kudu)
I20260929 18:41:21.332474 30835 webserver.cc:460] Webserver started at http://127.13.100.1:43179/ using document root <none> and password file <none>
I20260929 18:41:21.333043 30835 fs_manager.cc:359] Metadata directory not provided
I20260929 18:41:21.333133 30835 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:41:21.333361 30835 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:41:21.335443 30835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707036041078-13712-0/minicluster-data/ts-0/data/instance:
uuid: "24b6f50bface46e69c3ee7f7f75a2d49"
format_stamp: "Formatted at 2026-09-29 18:41:21 on dist-test-slave-j1xs"
server_key: "428246362b12995e060df2dd90213cd5"
server_key_iv: "6e1cac1e86434def1cd83a8dffc784ca"
server_key_version: "encryptionkey@0"
I20260929 18:41:21.336012 30835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707036041078-13712-0/minicluster-data/ts-0/wal/instance:
uuid: "24b6f50bface46e69c3ee7f7f75a2d49"
format_stamp: "Formatted at 2026-09-29 18:41:21 on dist-test-slave-j1xs"
server_key: "428246362b12995e060df2dd90213cd5"
server_key_iv: "6e1cac1e86434def1cd83a8dffc784ca"
server_key_version: "encryptionkey@0"
I20260929 18:41:21.339809 30835 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260929 18:41:21.342386 30850 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:41:21.343633 30835 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:41:21.343779 30835 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707036041078-13712-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707036041078-13712-0/minicluster-data/ts-0/wal
uuid: "24b6f50bface46e69c3ee7f7f75a2d49"
format_stamp: "Formatted at 2026-09-29 18:41:21 on dist-test-slave-j1xs"
server_key: "428246362b12995e060df2dd90213cd5"
server_key_iv: "6e1cac1e86434def1cd83a8dffc784ca"
server_key_version: "encryptionkey@0"
I20260929 18:41:21.343899 30835 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707036041078-13712-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707036041078-13712-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707036041078-13712-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:41:21.365547 30835 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:41:21.368898 30835 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:41:21.369107 30835 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:41:21.369695 30835 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:41:21.370522 30835 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:41:21.370594 30835 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:41:21.370663 30835 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:41:21.370708 30835 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:41:21.379518 30835 rpc_server.cc:307] RPC server started. Bound to: 127.13.100.1:44151
I20260929 18:41:21.379561 30963 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.100.1:44151 every 8 connection(s)
I20260929 18:41:21.380429 30835 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707036041078-13712-0/minicluster-data/ts-0/data/info.pb
I20260929 18:41:21.389573 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 30835
I20260929 18:41:21.389683 13712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707036041078-13712-0/minicluster-data/ts-0/wal/instance
I20260929 18:41:21.390038 13712 external_mini_cluster.cc:1428] Setting key 68a86c1c0138b3742c27d8f7ba0b16ff
Sep 29 18:41:21 dist-test-slave-j1xs krb5kdc[30728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707281, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.1@KRBTEST.COM for kudu/127.13.100.62@KRBTEST.COM
I20260929 18:41:21.394397 30812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:21.382451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.1:49239 (local address 127.13.100.62:40053)
0929 18:41:21.382604 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0929 18:41:21.382608 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:41:21.383406 (+ 798us) server_negotiation.cc:407] Connection header received
0929 18:41:21.384759 (+ 1353us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:41:21.384762 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:41:21.384807 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:41:21.384894 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:41:21.386010 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:21.386581 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:21.387658 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:21.387881 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:21.387961 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:21.390769 (+ 2808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:41:21.390773 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:41:21.390776 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:41:21.390810 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:41:21.392288 (+ 1478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:41:21.392809 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:41:21.392813 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:41:21.392816 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:41:21.392903 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:41:21.393253 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:41:21.393258 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:41:21.393260 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:41:21.393447 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:41:21.393556 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 18:41:21.394102 (+ 546us) server_negotiation.cc:299] Negotiation successful
0929 18:41:21.394224 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 18:41:21.395186 30966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:21.382796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:40053 (local address 127.13.100.1:49239)
0929 18:41:21.383249 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:21.383291 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 18:41:21.384554 (+ 1263us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:41:21.385041 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:41:21.385048 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:41:21.385401 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:41:21.385857 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:41:21.385872 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:41:21.386689 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:41:21.386691 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:41:21.387548 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:41:21.387557 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:41:21.388010 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:41:21.388012 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:41:21.388141 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:21.388797 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:41:21.388813 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:41:21.390589 (+ 1776us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:41:21.392425 (+ 1836us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:41:21.392430 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:41:21.392444 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:41:21.392702 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:41:21.392998 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:41:21.393003 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:41:21.393005 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:41:21.393154 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:41:21.393554 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:41:21.393561 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:41:21.393800 (+ 239us) client_negotiation.cc:769] Sending connection context
0929 18:41:21.394049 (+ 249us) client_negotiation.cc:240] Negotiation successful
0929 18:41:21.394368 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":108,"threads_started":1}
I20260929 18:41:21.396421 30964 heartbeater.cc:347] Connected to a master server at 127.13.100.62:40053
I20260929 18:41:21.396661 30964 heartbeater.cc:464] Registering TS with master...
I20260929 18:41:21.397131 30964 heartbeater.cc:511] Master 127.13.100.62:40053 requested a full tablet report, sending...
I20260929 18:41:21.398844 30773 ts_manager.cc:194] Registered new tserver with Master: 24b6f50bface46e69c3ee7f7f75a2d49 (127.13.100.1:44151)
WARNING: no policy specified for kudu/127.13.100.2@KRBTEST.COM; defaulting to no policy
I20260929 18:41:21.400180 30773 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.100.1@KRBTEST.COM'} at 127.13.100.1:49239
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.100.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.100.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.2.keytab.
Entry for principal kudu/127.13.100.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.2.keytab.
I20260929 18:41:21.417187 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707036041078-13712-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707036041078-13712-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707036041078-13712-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707036041078-13712-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.100.2:0
--local_ip_for_outbound_sockets=127.13.100.2
--webserver_interface=127.13.100.2
--webserver_port=0
--tserver_master_addrs=127.13.100.62:40053
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.2.keytab
--principal=kudu/127.13.100.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.100.20:32911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707036041078-13712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707036041078-13712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707036041078-13712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:41:21.528183 30969 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:41:21.528414 30969 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:41:21.528455 30969 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:41:21.532374 30969 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:41:21.532428 30969 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:41:21.532485 30969 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.100.2
I20260929 18:41:21.537603 30969 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.100.20:32911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707036041078-13712-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707036041078-13712-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.2.keytab
--principal=kudu/127.13.100.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.100.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707036041078-13712-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.100.2
--webserver_port=0
--tserver_master_addrs=127.13.100.62:40053
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30969
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.100.2
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707036041078-13712-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:41:21.538635 30969 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:41:21.539613 30969 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:41:21.549026 30974 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:41:21.548965 30975 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:41:21.548950 30977 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:41:21.549449 30969 server_base.cc:1034] running on GCE node
I20260929 18:41:21.549825 30969 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:41:21.550498 30969 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:41:21.551652 30969 hybrid_clock.cc:648] HybridClock initialized: now 1790707281551623 us; error 51 us; skew 500 ppm
Sep 29 18:41:21 dist-test-slave-j1xs krb5kdc[30728](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707281, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:41:21.554334 30969 init.cc:377] Logged in from keytab as kudu/127.13.100.2@KRBTEST.COM (short username kudu)
I20260929 18:41:21.555171 30969 webserver.cc:460] Webserver started at http://127.13.100.2:42059/ using document root <none> and password file <none>
I20260929 18:41:21.555716 30969 fs_manager.cc:359] Metadata directory not provided
I20260929 18:41:21.555776 30969 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:41:21.555980 30969 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:41:21.557988 30969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707036041078-13712-0/minicluster-data/ts-1/data/instance:
uuid: "1582c75505e0403ebfaefca76ccecf35"
format_stamp: "Formatted at 2026-09-29 18:41:21 on dist-test-slave-j1xs"
server_key: "0795dcb8d94d2e5c577729cfd1757b2a"
server_key_iv: "e1bda51ef97bd897f8e4b5edb21d04e5"
server_key_version: "encryptionkey@0"
I20260929 18:41:21.558544 30969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707036041078-13712-0/minicluster-data/ts-1/wal/instance:
uuid: "1582c75505e0403ebfaefca76ccecf35"
format_stamp: "Formatted at 2026-09-29 18:41:21 on dist-test-slave-j1xs"
server_key: "0795dcb8d94d2e5c577729cfd1757b2a"
server_key_iv: "e1bda51ef97bd897f8e4b5edb21d04e5"
server_key_version: "encryptionkey@0"
I20260929 18:41:21.562211 30969 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20260929 18:41:21.564579 30984 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:41:21.565786 30969 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:41:21.565943 30969 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707036041078-13712-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707036041078-13712-0/minicluster-data/ts-1/wal
uuid: "1582c75505e0403ebfaefca76ccecf35"
format_stamp: "Formatted at 2026-09-29 18:41:21 on dist-test-slave-j1xs"
server_key: "0795dcb8d94d2e5c577729cfd1757b2a"
server_key_iv: "e1bda51ef97bd897f8e4b5edb21d04e5"
server_key_version: "encryptionkey@0"
I20260929 18:41:21.566056 30969 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707036041078-13712-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707036041078-13712-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707036041078-13712-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:41:21.577921 30969 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:41:21.580940 30969 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:41:21.581135 30969 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:41:21.581722 30969 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:41:21.582532 30969 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:41:21.582578 30969 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:41:21.582617 30969 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:41:21.582633 30969 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:41:21.592176 30969 rpc_server.cc:307] RPC server started. Bound to: 127.13.100.2:41735
I20260929 18:41:21.592206 31097 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.100.2:41735 every 8 connection(s)
I20260929 18:41:21.593045 30969 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707036041078-13712-0/minicluster-data/ts-1/data/info.pb
I20260929 18:41:21.593616 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 30969
I20260929 18:41:21.593758 13712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707036041078-13712-0/minicluster-data/ts-1/wal/instance
I20260929 18:41:21.594107 13712 external_mini_cluster.cc:1428] Setting key 2dbff692f36704767d5d03e5fb5f5100
Sep 29 18:41:21 dist-test-slave-j1xs krb5kdc[30728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707281, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.2@KRBTEST.COM for kudu/127.13.100.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.13.100.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.100.3@KRBTEST.COM" created.
I20260929 18:41:21.608407 30812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:21.595049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.2:36535 (local address 127.13.100.62:40053)
0929 18:41:21.595212 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0929 18:41:21.595216 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:41:21.596154 (+ 938us) server_negotiation.cc:407] Connection header received
0929 18:41:21.597975 (+ 1821us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:41:21.597980 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:41:21.598038 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:41:21.598136 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:41:21.599489 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:21.600048 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:21.601867 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:21.602139 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:21.602216 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:21.604960 (+ 2744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:41:21.604963 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:41:21.604965 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:41:21.604988 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:41:21.606360 (+ 1372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:41:21.606934 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:41:21.606936 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:41:21.606937 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:41:21.607008 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:41:21.607319 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:41:21.607322 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:41:21.607323 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:41:21.607491 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:41:21.607596 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 18:41:21.608187 (+ 591us) server_negotiation.cc:299] Negotiation successful
0929 18:41:21.608281 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 18:41:21.609573 31100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:21.595435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:40053 (local address 127.13.100.2:36535)
0929 18:41:21.595982 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:21.596038 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0929 18:41:21.597732 (+ 1694us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:41:21.598278 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:41:21.598284 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:41:21.598616 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:41:21.599323 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:41:21.599351 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:41:21.600180 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:41:21.600184 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:41:21.601725 (+ 1541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:41:21.601739 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:41:21.602289 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:41:21.602293 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:41:21.602501 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:21.603141 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:41:21.603167 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:41:21.604797 (+ 1630us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:41:21.606465 (+ 1668us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:41:21.606469 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:41:21.606479 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:41:21.606834 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:41:21.607113 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:41:21.607115 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:41:21.607116 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:41:21.607236 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:41:21.607629 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:41:21.607634 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:41:21.607925 (+ 291us) client_negotiation.cc:769] Sending connection context
0929 18:41:21.608141 (+ 216us) client_negotiation.cc:240] Negotiation successful
0929 18:41:21.608496 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":135,"threads_started":1}
I20260929 18:41:21.610910 31098 heartbeater.cc:347] Connected to a master server at 127.13.100.62:40053
I20260929 18:41:21.611172 31098 heartbeater.cc:464] Registering TS with master...
I20260929 18:41:21.611704 31098 heartbeater.cc:511] Master 127.13.100.62:40053 requested a full tablet report, sending...
I20260929 18:41:21.612903 30773 ts_manager.cc:194] Registered new tserver with Master: 1582c75505e0403ebfaefca76ccecf35 (127.13.100.2:41735)
I20260929 18:41:21.613785 30773 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.100.2@KRBTEST.COM'} at 127.13.100.2:36535
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.100.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.3.keytab.
Entry for principal kudu/127.13.100.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.3.keytab.
I20260929 18:41:21.624641 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707036041078-13712-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707036041078-13712-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707036041078-13712-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707036041078-13712-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.100.3:0
--local_ip_for_outbound_sockets=127.13.100.3
--webserver_interface=127.13.100.3
--webserver_port=0
--tserver_master_addrs=127.13.100.62:40053
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.3.keytab
--principal=kudu/127.13.100.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.100.20:32911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707036041078-13712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707036041078-13712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707036041078-13712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:41:21.737231 31103 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:41:21.737504 31103 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:41:21.737550 31103 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:41:21.741576 31103 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:41:21.741636 31103 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:41:21.741695 31103 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.100.3
I20260929 18:41:21.747231 31103 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.100.20:32911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707036041078-13712-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707036041078-13712-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.3.keytab
--principal=kudu/127.13.100.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.100.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707036041078-13712-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.100.3
--webserver_port=0
--tserver_master_addrs=127.13.100.62:40053
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31103
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.100.3
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707036041078-13712-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:41:21.748255 31103 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:41:21.749280 31103 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:41:21.758812 31111 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:41:21.758843 31109 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:41:21.758812 31108 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:41:21.758992 31103 server_base.cc:1034] running on GCE node
I20260929 18:41:21.759544 31103 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:41:21.760231 31103 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:41:21.761422 31103 hybrid_clock.cc:648] HybridClock initialized: now 1790707281761413 us; error 33 us; skew 500 ppm
Sep 29 18:41:21 dist-test-slave-j1xs krb5kdc[30728](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707281, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:41:21.764325 31103 init.cc:377] Logged in from keytab as kudu/127.13.100.3@KRBTEST.COM (short username kudu)
I20260929 18:41:21.765174 31103 webserver.cc:460] Webserver started at http://127.13.100.3:44483/ using document root <none> and password file <none>
I20260929 18:41:21.765681 31103 fs_manager.cc:359] Metadata directory not provided
I20260929 18:41:21.765740 31103 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:41:21.765980 31103 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:41:21.768478 31103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707036041078-13712-0/minicluster-data/ts-2/data/instance:
uuid: "c0e3180d7fe243ac93f24e580793b6ca"
format_stamp: "Formatted at 2026-09-29 18:41:21 on dist-test-slave-j1xs"
server_key: "27ad64cc54857489757d237807cc8d39"
server_key_iv: "422b26f67f2df8bae988723681eab242"
server_key_version: "encryptionkey@0"
I20260929 18:41:21.769074 31103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707036041078-13712-0/minicluster-data/ts-2/wal/instance:
uuid: "c0e3180d7fe243ac93f24e580793b6ca"
format_stamp: "Formatted at 2026-09-29 18:41:21 on dist-test-slave-j1xs"
server_key: "27ad64cc54857489757d237807cc8d39"
server_key_iv: "422b26f67f2df8bae988723681eab242"
server_key_version: "encryptionkey@0"
I20260929 18:41:21.772840 31103 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260929 18:41:21.775322 31118 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:41:21.776557 31103 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 18:41:21.776654 31103 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707036041078-13712-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707036041078-13712-0/minicluster-data/ts-2/wal
uuid: "c0e3180d7fe243ac93f24e580793b6ca"
format_stamp: "Formatted at 2026-09-29 18:41:21 on dist-test-slave-j1xs"
server_key: "27ad64cc54857489757d237807cc8d39"
server_key_iv: "422b26f67f2df8bae988723681eab242"
server_key_version: "encryptionkey@0"
I20260929 18:41:21.776736 31103 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707036041078-13712-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707036041078-13712-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707036041078-13712-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:41:21.787696 31103 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:41:21.790798 31103 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:41:21.790954 31103 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:41:21.791467 31103 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:41:21.792207 31103 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:41:21.792253 31103 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:41:21.792292 31103 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:41:21.792308 31103 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:41:21.801874 31103 rpc_server.cc:307] RPC server started. Bound to: 127.13.100.3:35519
I20260929 18:41:21.801962 31231 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.100.3:35519 every 8 connection(s)
I20260929 18:41:21.802809 31103 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707036041078-13712-0/minicluster-data/ts-2/data/info.pb
I20260929 18:41:21.811138 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 31103
I20260929 18:41:21.811313 13712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707036041078-13712-0/minicluster-data/ts-2/wal/instance
I20260929 18:41:21.811628 13712 external_mini_cluster.cc:1428] Setting key 0d874ee67eaf5ea35f5709522de6a713
Sep 29 18:41:21 dist-test-slave-j1xs krb5kdc[30728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707281, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.3@KRBTEST.COM for kudu/127.13.100.62@KRBTEST.COM
I20260929 18:41:21.817013 30812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:21.804719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.3:53371 (local address 127.13.100.62:40053)
0929 18:41:21.804858 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0929 18:41:21.804862 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:41:21.805638 (+ 776us) server_negotiation.cc:407] Connection header received
0929 18:41:21.806987 (+ 1349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:41:21.806990 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:41:21.807053 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:41:21.807151 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:41:21.808306 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:21.808832 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:21.809999 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:21.810219 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:21.810329 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:21.813229 (+ 2900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:41:21.813233 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:41:21.813236 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:41:21.813264 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:41:21.814888 (+ 1624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:41:21.815423 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:41:21.815426 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:41:21.815428 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:41:21.815539 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:41:21.815949 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:41:21.815952 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:41:21.815953 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:41:21.816141 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:41:21.816228 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 18:41:21.816777 (+ 549us) server_negotiation.cc:299] Negotiation successful
0929 18:41:21.816883 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 18:41:21.817796 31234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:21.805061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:40053 (local address 127.13.100.3:53371)
0929 18:41:21.805490 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:21.805530 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 18:41:21.806769 (+ 1239us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:41:21.807320 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:41:21.807327 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:41:21.807684 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:41:21.808146 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:41:21.808159 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:41:21.808963 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:41:21.808967 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:41:21.809842 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:41:21.809852 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:41:21.810375 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:41:21.810379 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:41:21.810550 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:21.811208 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:41:21.811225 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:41:21.813060 (+ 1835us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:41:21.814995 (+ 1935us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:41:21.815000 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:41:21.815011 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:41:21.815269 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:41:21.815648 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:41:21.815651 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:41:21.815653 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:41:21.815791 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:41:21.816255 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:41:21.816260 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:41:21.816512 (+ 252us) client_negotiation.cc:769] Sending connection context
0929 18:41:21.816678 (+ 166us) client_negotiation.cc:240] Negotiation successful
0929 18:41:21.816942 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":107,"threads_started":1}
I20260929 18:41:21.818951 31232 heartbeater.cc:347] Connected to a master server at 127.13.100.62:40053
I20260929 18:41:21.819213 31232 heartbeater.cc:464] Registering TS with master...
I20260929 18:41:21.819749 31232 heartbeater.cc:511] Master 127.13.100.62:40053 requested a full tablet report, sending...
I20260929 18:41:21.820755 30773 ts_manager.cc:194] Registered new tserver with Master: c0e3180d7fe243ac93f24e580793b6ca (127.13.100.3:35519)
I20260929 18:41:21.821321 30773 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.100.3@KRBTEST.COM'} at 127.13.100.3:53371
I20260929 18:41:21.825970 13712 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 18:41:21.835561 30812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:21.828170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56904 (local address 127.13.100.62:40053)
0929 18:41:21.828380 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0929 18:41:21.828383 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:41:21.828403 (+ 20us) server_negotiation.cc:407] Connection header received
0929 18:41:21.828511 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:41:21.828514 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:41:21.828574 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:41:21.828684 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:41:21.829363 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:21.829934 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:21.830771 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:21.831010 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:21.831117 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:21.832512 (+ 1395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:41:21.832515 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:41:21.832517 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:41:21.832541 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:41:21.833973 (+ 1432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:41:21.834348 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:41:21.834350 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:41:21.834352 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:41:21.834425 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:41:21.834667 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:41:21.834670 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:41:21.834671 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:41:21.834889 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:41:21.834992 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 18:41:21.835329 (+ 337us) server_negotiation.cc:299] Negotiation successful
0929 18:41:21.835438 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121}
I20260929 18:41:21.838680 30773 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56904:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:41:21.841694 30773 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:41:21.856226 31247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:21.850870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.3:35519 (local address 127.0.0.1:51860)
0929 18:41:21.851662 (+ 792us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:21.851677 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 18:41:21.851764 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:41:21.852270 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:41:21.852273 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:41:21.852292 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:41:21.852391 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:41:21.852397 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:41:21.853515 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:41:21.853518 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:41:21.854714 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:41:21.854723 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:41:21.855666 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:41:21.855670 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:41:21.855773 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:21.855820 (+ 47us) client_negotiation.cc:769] Sending connection context
0929 18:41:21.855945 (+ 125us) client_negotiation.cc:240] Negotiation successful
0929 18:41:21.856074 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":727,"mutex_wait_us":227,"thread_start_us":83,"threads_started":1}
I20260929 18:41:21.856240 31248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:21.851244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51860 (local address 127.13.100.3:35519)
0929 18:41:21.851651 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0929 18:41:21.851663 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 18:41:21.851741 (+ 78us) server_negotiation.cc:407] Connection header received
0929 18:41:21.851903 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:41:21.851908 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:41:21.852112 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:41:21.852248 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:41:21.852510 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:21.853231 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:21.854840 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:21.855534 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:21.855641 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:21.855779 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0929 18:41:21.855989 (+ 210us) server_negotiation.cc:299] Negotiation successful
0929 18:41:21.856086 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":169,"threads_started":1}
I20260929 18:41:21.856742 31243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:21.850217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.2:41735 (local address 127.0.0.1:38606)
0929 18:41:21.850947 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:21.850998 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 18:41:21.851113 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:41:21.851954 (+ 841us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:41:21.851958 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:41:21.851976 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:41:21.852117 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:41:21.852123 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:41:21.853641 (+ 1518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:41:21.853644 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:41:21.855130 (+ 1486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:41:21.855141 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:41:21.856352 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:41:21.856355 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:41:21.856449 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:21.856472 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 18:41:21.856539 (+ 67us) client_negotiation.cc:240] Negotiation successful
0929 18:41:21.856609 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"mutex_wait_us":184,"thread_start_us":61,"threads_started":1}
I20260929 18:41:21.856951 31246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:21.850692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34248 (local address 127.13.100.1:44151)
0929 18:41:21.851068 (+ 376us) server_negotiation.cc:206] Beginning negotiation
0929 18:41:21.851084 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 18:41:21.851661 (+ 577us) server_negotiation.cc:407] Connection header received
0929 18:41:21.851753 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:41:21.851761 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:41:21.851938 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:41:21.852078 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:41:21.852291 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:21.853488 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:21.855270 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:21.855953 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:21.856041 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:21.856162 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0929 18:41:21.856711 (+ 549us) server_negotiation.cc:299] Negotiation successful
0929 18:41:21.856813 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":82,"threads_started":1}
I20260929 18:41:21.857144 31245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:21.850333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38606 (local address 127.13.100.2:41735)
0929 18:41:21.851270 (+ 937us) server_negotiation.cc:206] Beginning negotiation
0929 18:41:21.851281 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 18:41:21.851303 (+ 22us) server_negotiation.cc:407] Connection header received
0929 18:41:21.851382 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:41:21.851390 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:41:21.851601 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:41:21.851761 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:41:21.852282 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:21.853481 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:21.855257 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:21.856057 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:21.856157 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:21.856282 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0929 18:41:21.856924 (+ 642us) server_negotiation.cc:299] Negotiation successful
0929 18:41:21.857015 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":843,"thread_start_us":111,"threads_started":1}
I20260929 18:41:21.857436 31244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:21.850326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.1:44151 (local address 127.0.0.1:34248)
0929 18:41:21.851452 (+ 1126us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:21.851469 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 18:41:21.851562 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:41:21.852073 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:41:21.852076 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:41:21.852088 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:41:21.852155 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:41:21.852164 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:41:21.853641 (+ 1477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:41:21.853644 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:41:21.855130 (+ 1486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:41:21.855141 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:41:21.856218 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:41:21.856221 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:41:21.856319 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:21.856348 (+ 29us) client_negotiation.cc:769] Sending connection context
0929 18:41:21.856444 (+ 96us) client_negotiation.cc:240] Negotiation successful
0929 18:41:21.857312 (+ 868us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":964,"mutex_wait_us":94,"thread_start_us":120,"threads_started":1}
I20260929 18:41:21.859675 30898 tablet_service.cc:1467] Processing CreateTablet for tablet e1648bca6d2f45bb9e3b02510e8ed4e6 (DEFAULT_TABLE table=test-table [id=ea0dc243a33c41da8dd1a56df074a5d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:41:21.859611 31166 tablet_service.cc:1467] Processing CreateTablet for tablet e1648bca6d2f45bb9e3b02510e8ed4e6 (DEFAULT_TABLE table=test-table [id=ea0dc243a33c41da8dd1a56df074a5d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:41:21.860116 31032 tablet_service.cc:1467] Processing CreateTablet for tablet e1648bca6d2f45bb9e3b02510e8ed4e6 (DEFAULT_TABLE table=test-table [id=ea0dc243a33c41da8dd1a56df074a5d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:41:21.860657 30898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1648bca6d2f45bb9e3b02510e8ed4e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:41:21.860671 31166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1648bca6d2f45bb9e3b02510e8ed4e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:41:21.860978 31032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1648bca6d2f45bb9e3b02510e8ed4e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:41:21.866633 31249 tablet_bootstrap.cc:492] T e1648bca6d2f45bb9e3b02510e8ed4e6 P 24b6f50bface46e69c3ee7f7f75a2d49: Bootstrap starting.
I20260929 18:41:21.867353 31250 tablet_bootstrap.cc:492] T e1648bca6d2f45bb9e3b02510e8ed4e6 P c0e3180d7fe243ac93f24e580793b6ca: Bootstrap starting.
I20260929 18:41:21.868192 31251 tablet_bootstrap.cc:492] T e1648bca6d2f45bb9e3b02510e8ed4e6 P 1582c75505e0403ebfaefca76ccecf35: Bootstrap starting.
I20260929 18:41:21.869294 31250 tablet_bootstrap.cc:654] T e1648bca6d2f45bb9e3b02510e8ed4e6 P c0e3180d7fe243ac93f24e580793b6ca: Neither blocks nor log segments found. Creating new log.
I20260929 18:41:21.869294 31249 tablet_bootstrap.cc:654] T e1648bca6d2f45bb9e3b02510e8ed4e6 P 24b6f50bface46e69c3ee7f7f75a2d49: Neither blocks nor log segments found. Creating new log.
I20260929 18:41:21.870513 31251 tablet_bootstrap.cc:654] T e1648bca6d2f45bb9e3b02510e8ed4e6 P 1582c75505e0403ebfaefca76ccecf35: Neither blocks nor log segments found. Creating new log.
I20260929 18:41:21.870643 31249 log.cc:824] T e1648bca6d2f45bb9e3b02510e8ed4e6 P 24b6f50bface46e69c3ee7f7f75a2d49: Log is configured to *not* fsync() on all Append() calls
I20260929 18:41:21.870643 31250 log.cc:824] T e1648bca6d2f45bb9e3b02510e8ed4e6 P c0e3180d7fe243ac93f24e580793b6ca: Log is configured to *not* fsync() on all Append() calls
I20260929 18:41:21.871397 31251 log.cc:824] T e1648bca6d2f45bb9e3b02510e8ed4e6 P 1582c75505e0403ebfaefca76ccecf35: Log is configured to *not* fsync() on all Append() calls
I20260929 18:41:21.872716 31249 tablet_bootstrap.cc:492] T e1648bca6d2f45bb9e3b02510e8ed4e6 P 24b6f50bface46e69c3ee7f7f75a2d49: No bootstrap required, opened a new log
I20260929 18:41:21.872923 31249 ts_tablet_manager.cc:1397] T e1648bca6d2f45bb9e3b02510e8ed4e6 P 24b6f50bface46e69c3ee7f7f75a2d49: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.003s
I20260929 18:41:21.873136 31251 tablet_bootstrap.cc:492] T e1648bca6d2f45bb9e3b02510e8ed4e6 P 1582c75505e0403ebfaefca76ccecf35: No bootstrap required, opened a new log
I20260929 18:41:21.873308 31251 ts_tablet_manager.cc:1397] T e1648bca6d2f45bb9e3b02510e8ed4e6 P 1582c75505e0403ebfaefca76ccecf35: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 18:41:21.875247 31250 tablet_bootstrap.cc:492] T e1648bca6d2f45bb9e3b02510e8ed4e6 P c0e3180d7fe243ac93f24e580793b6ca: No bootstrap required, opened a new log
I20260929 18:41:21.875433 31250 ts_tablet_manager.cc:1397] T e1648bca6d2f45bb9e3b02510e8ed4e6 P c0e3180d7fe243ac93f24e580793b6ca: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.004s
I20260929 18:41:21.876298 31251 raft_consensus.cc:348] T e1648bca6d2f45bb9e3b02510e8ed4e6 P 1582c75505e0403ebfaefca76ccecf35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1582c75505e0403ebfaefca76ccecf35" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 41735 } } peers { permanent_uuid: "24b6f50bface46e69c3ee7f7f75a2d49" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 44151 } } peers { permanent_uuid: "c0e3180d7fe243ac93f24e580793b6ca" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 35519 } }
I20260929 18:41:21.876546 31251 raft_consensus.cc:374] T e1648bca6d2f45bb9e3b02510e8ed4e6 P 1582c75505e0403ebfaefca76ccecf35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:41:21.876617 31251 raft_consensus.cc:729] T e1648bca6d2f45bb9e3b02510e8ed4e6 P 1582c75505e0403ebfaefca76ccecf35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1582c75505e0403ebfaefca76ccecf35, State: Initialized, Role: FOLLOWER
I20260929 18:41:21.876760 31249 raft_consensus.cc:348] T e1648bca6d2f45bb9e3b02510e8ed4e6 P 24b6f50bface46e69c3ee7f7f75a2d49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1582c75505e0403ebfaefca76ccecf35" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 41735 } } peers { permanent_uuid: "24b6f50bface46e69c3ee7f7f75a2d49" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 44151 } } peers { permanent_uuid: "c0e3180d7fe243ac93f24e580793b6ca" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 35519 } }
I20260929 18:41:21.877003 31249 raft_consensus.cc:374] T e1648bca6d2f45bb9e3b02510e8ed4e6 P 24b6f50bface46e69c3ee7f7f75a2d49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:41:21.877099 31249 raft_consensus.cc:729] T e1648bca6d2f45bb9e3b02510e8ed4e6 P 24b6f50bface46e69c3ee7f7f75a2d49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24b6f50bface46e69c3ee7f7f75a2d49, State: Initialized, Role: FOLLOWER
I20260929 18:41:21.877096 31251 consensus_queue.cc:260] T e1648bca6d2f45bb9e3b02510e8ed4e6 P 1582c75505e0403ebfaefca76ccecf35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1582c75505e0403ebfaefca76ccecf35" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 41735 } } peers { permanent_uuid: "24b6f50bface46e69c3ee7f7f75a2d49" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 44151 } } peers { permanent_uuid: "c0e3180d7fe243ac93f24e580793b6ca" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 35519 } }
I20260929 18:41:21.877779 31098 heartbeater.cc:503] Master 127.13.100.62:40053 was elected leader, sending a full tablet report...
I20260929 18:41:21.878115 31251 ts_tablet_manager.cc:1428] T e1648bca6d2f45bb9e3b02510e8ed4e6 P 1582c75505e0403ebfaefca76ccecf35: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260929 18:41:21.877557 31249 consensus_queue.cc:260] T e1648bca6d2f45bb9e3b02510e8ed4e6 P 24b6f50bface46e69c3ee7f7f75a2d49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1582c75505e0403ebfaefca76ccecf35" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 41735 } } peers { permanent_uuid: "24b6f50bface46e69c3ee7f7f75a2d49" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 44151 } } peers { permanent_uuid: "c0e3180d7fe243ac93f24e580793b6ca" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 35519 } }
I20260929 18:41:21.879320 30964 heartbeater.cc:503] Master 127.13.100.62:40053 was elected leader, sending a full tablet report...
I20260929 18:41:21.879168 31250 raft_consensus.cc:348] T e1648bca6d2f45bb9e3b02510e8ed4e6 P c0e3180d7fe243ac93f24e580793b6ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1582c75505e0403ebfaefca76ccecf35" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 41735 } } peers { permanent_uuid: "24b6f50bface46e69c3ee7f7f75a2d49" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 44151 } } peers { permanent_uuid: "c0e3180d7fe243ac93f24e580793b6ca" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 35519 } }
I20260929 18:41:21.879413 31250 raft_consensus.cc:374] T e1648bca6d2f45bb9e3b02510e8ed4e6 P c0e3180d7fe243ac93f24e580793b6ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:41:21.879472 31250 raft_consensus.cc:729] T e1648bca6d2f45bb9e3b02510e8ed4e6 P c0e3180d7fe243ac93f24e580793b6ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0e3180d7fe243ac93f24e580793b6ca, State: Initialized, Role: FOLLOWER
I20260929 18:41:21.879637 31249 ts_tablet_manager.cc:1428] T e1648bca6d2f45bb9e3b02510e8ed4e6 P 24b6f50bface46e69c3ee7f7f75a2d49: Time spent starting tablet: real 0.007s user 0.004s sys 0.002s
I20260929 18:41:21.879951 31250 consensus_queue.cc:260] T e1648bca6d2f45bb9e3b02510e8ed4e6 P c0e3180d7fe243ac93f24e580793b6ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1582c75505e0403ebfaefca76ccecf35" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 41735 } } peers { permanent_uuid: "24b6f50bface46e69c3ee7f7f75a2d49" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 44151 } } peers { permanent_uuid: "c0e3180d7fe243ac93f24e580793b6ca" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 35519 } }
I20260929 18:41:21.881012 31250 ts_tablet_manager.cc:1428] T e1648bca6d2f45bb9e3b02510e8ed4e6 P c0e3180d7fe243ac93f24e580793b6ca: Time spent starting tablet: real 0.006s user 0.005s sys 0.002s
I20260929 18:41:21.881362 31232 heartbeater.cc:503] Master 127.13.100.62:40053 was elected leader, sending a full tablet report...
W20260929 18:41:21.881947 30965 tablet.cc:2369] T e1648bca6d2f45bb9e3b02510e8ed4e6 P 24b6f50bface46e69c3ee7f7f75a2d49: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:41:21.924286 31257 raft_consensus.cc:482] T e1648bca6d2f45bb9e3b02510e8ed4e6 P c0e3180d7fe243ac93f24e580793b6ca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:41:21.924564 31257 raft_consensus.cc:504] T e1648bca6d2f45bb9e3b02510e8ed4e6 P c0e3180d7fe243ac93f24e580793b6ca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1582c75505e0403ebfaefca76ccecf35" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 41735 } } peers { permanent_uuid: "24b6f50bface46e69c3ee7f7f75a2d49" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 44151 } } peers { permanent_uuid: "c0e3180d7fe243ac93f24e580793b6ca" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 35519 } }
I20260929 18:41:21.925787 31257 leader_election.cc:288] T e1648bca6d2f45bb9e3b02510e8ed4e6 P c0e3180d7fe243ac93f24e580793b6ca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1582c75505e0403ebfaefca76ccecf35 (127.13.100.2:41735), 24b6f50bface46e69c3ee7f7f75a2d49 (127.13.100.1:44151)
I20260929 18:41:21.930682 31246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:21.926261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.3:33333 (local address 127.13.100.1:44151)
0929 18:41:21.926426 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0929 18:41:21.926430 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:41:21.926447 (+ 17us) server_negotiation.cc:407] Connection header received
0929 18:41:21.926608 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:41:21.926618 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:41:21.926693 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:41:21.926796 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:41:21.927093 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:21.927736 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:21.929349 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:21.930017 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:21.930156 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:21.930217 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0929 18:41:21.930451 (+ 234us) server_negotiation.cc:299] Negotiation successful
0929 18:41:21.930547 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 18:41:21.930646 31234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:21.926132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.1:44151 (local address 127.13.100.3:33333)
0929 18:41:21.926331 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:21.926347 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 18:41:21.926485 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:41:21.926812 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:41:21.926815 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:41:21.926831 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:41:21.926950 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:41:21.926955 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:41:21.927855 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:41:21.927870 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:41:21.929199 (+ 1329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:41:21.929210 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:41:21.930152 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:41:21.930155 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:41:21.930307 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:21.930333 (+ 26us) client_negotiation.cc:769] Sending connection context
0929 18:41:21.930415 (+ 82us) client_negotiation.cc:240] Negotiation successful
0929 18:41:21.930493 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"mutex_wait_us":46}
I20260929 18:41:21.930815 31245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:21.926137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.3:54153 (local address 127.13.100.2:41735)
0929 18:41:21.926279 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0929 18:41:21.926284 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:41:21.926686 (+ 402us) server_negotiation.cc:407] Connection header received
0929 18:41:21.926902 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:41:21.926905 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:41:21.926951 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:41:21.927041 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:41:21.927348 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:21.928089 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:21.929536 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:21.930211 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:21.930336 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:21.930380 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0929 18:41:21.930587 (+ 207us) server_negotiation.cc:299] Negotiation successful
0929 18:41:21.930674 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 18:41:21.930832 31258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:21.926164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.2:41735 (local address 127.13.100.3:54153)
0929 18:41:21.926531 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:21.926584 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 18:41:21.926694 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:41:21.927071 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:41:21.927074 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:41:21.927089 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:41:21.927249 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:41:21.927257 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:41:21.928226 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:41:21.928230 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:41:21.929403 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:41:21.929410 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:41:21.930336 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:41:21.930339 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:41:21.930469 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:21.930488 (+ 19us) client_negotiation.cc:769] Sending connection context
0929 18:41:21.930558 (+ 70us) client_negotiation.cc:240] Negotiation successful
0929 18:41:21.930610 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"mutex_wait_us":23,"thread_start_us":124,"threads_started":1}
I20260929 18:41:21.931450 31052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1648bca6d2f45bb9e3b02510e8ed4e6" candidate_uuid: "c0e3180d7fe243ac93f24e580793b6ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1582c75505e0403ebfaefca76ccecf35" is_pre_election: true
I20260929 18:41:21.931523 30918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1648bca6d2f45bb9e3b02510e8ed4e6" candidate_uuid: "c0e3180d7fe243ac93f24e580793b6ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24b6f50bface46e69c3ee7f7f75a2d49" is_pre_election: true
I20260929 18:41:21.931774 31052 raft_consensus.cc:2454] T e1648bca6d2f45bb9e3b02510e8ed4e6 P 1582c75505e0403ebfaefca76ccecf35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0e3180d7fe243ac93f24e580793b6ca in term 0.
I20260929 18:41:21.931805 30918 raft_consensus.cc:2454] T e1648bca6d2f45bb9e3b02510e8ed4e6 P 24b6f50bface46e69c3ee7f7f75a2d49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0e3180d7fe243ac93f24e580793b6ca in term 0.
I20260929 18:41:21.932288 31121 leader_election.cc:302] T e1648bca6d2f45bb9e3b02510e8ed4e6 P c0e3180d7fe243ac93f24e580793b6ca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24b6f50bface46e69c3ee7f7f75a2d49, c0e3180d7fe243ac93f24e580793b6ca; no voters:
I20260929 18:41:21.932596 31257 raft_consensus.cc:2781] T e1648bca6d2f45bb9e3b02510e8ed4e6 P c0e3180d7fe243ac93f24e580793b6ca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:41:21.932685 31257 raft_consensus.cc:482] T e1648bca6d2f45bb9e3b02510e8ed4e6 P c0e3180d7fe243ac93f24e580793b6ca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:41:21.932726 31257 raft_consensus.cc:3037] T e1648bca6d2f45bb9e3b02510e8ed4e6 P c0e3180d7fe243ac93f24e580793b6ca [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:41:21.933959 31257 raft_consensus.cc:504] T e1648bca6d2f45bb9e3b02510e8ed4e6 P c0e3180d7fe243ac93f24e580793b6ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1582c75505e0403ebfaefca76ccecf35" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 41735 } } peers { permanent_uuid: "24b6f50bface46e69c3ee7f7f75a2d49" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 44151 } } peers { permanent_uuid: "c0e3180d7fe243ac93f24e580793b6ca" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 35519 } }
I20260929 18:41:21.934324 31257 leader_election.cc:288] T e1648bca6d2f45bb9e3b02510e8ed4e6 P c0e3180d7fe243ac93f24e580793b6ca [CANDIDATE]: Term 1 election: Requested vote from peers 1582c75505e0403ebfaefca76ccecf35 (127.13.100.2:41735), 24b6f50bface46e69c3ee7f7f75a2d49 (127.13.100.1:44151)
I20260929 18:41:21.934669 30918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1648bca6d2f45bb9e3b02510e8ed4e6" candidate_uuid: "c0e3180d7fe243ac93f24e580793b6ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24b6f50bface46e69c3ee7f7f75a2d49"
I20260929 18:41:21.934718 31052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1648bca6d2f45bb9e3b02510e8ed4e6" candidate_uuid: "c0e3180d7fe243ac93f24e580793b6ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1582c75505e0403ebfaefca76ccecf35"
I20260929 18:41:21.934803 30918 raft_consensus.cc:3037] T e1648bca6d2f45bb9e3b02510e8ed4e6 P 24b6f50bface46e69c3ee7f7f75a2d49 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:41:21.934873 31052 raft_consensus.cc:3037] T e1648bca6d2f45bb9e3b02510e8ed4e6 P 1582c75505e0403ebfaefca76ccecf35 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:41:21.935783 30918 raft_consensus.cc:2454] T e1648bca6d2f45bb9e3b02510e8ed4e6 P 24b6f50bface46e69c3ee7f7f75a2d49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0e3180d7fe243ac93f24e580793b6ca in term 1.
I20260929 18:41:21.936028 31052 raft_consensus.cc:2454] T e1648bca6d2f45bb9e3b02510e8ed4e6 P 1582c75505e0403ebfaefca76ccecf35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0e3180d7fe243ac93f24e580793b6ca in term 1.
I20260929 18:41:21.936105 31121 leader_election.cc:302] T e1648bca6d2f45bb9e3b02510e8ed4e6 P c0e3180d7fe243ac93f24e580793b6ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24b6f50bface46e69c3ee7f7f75a2d49, c0e3180d7fe243ac93f24e580793b6ca; no voters:
I20260929 18:41:21.936273 31257 raft_consensus.cc:2781] T e1648bca6d2f45bb9e3b02510e8ed4e6 P c0e3180d7fe243ac93f24e580793b6ca [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:41:21.936506 31257 raft_consensus.cc:686] T e1648bca6d2f45bb9e3b02510e8ed4e6 P c0e3180d7fe243ac93f24e580793b6ca [term 1 LEADER]: Becoming Leader. State: Replica: c0e3180d7fe243ac93f24e580793b6ca, State: Running, Role: LEADER
I20260929 18:41:21.936841 31257 consensus_queue.cc:237] T e1648bca6d2f45bb9e3b02510e8ed4e6 P c0e3180d7fe243ac93f24e580793b6ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1582c75505e0403ebfaefca76ccecf35" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 41735 } } peers { permanent_uuid: "24b6f50bface46e69c3ee7f7f75a2d49" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 44151 } } peers { permanent_uuid: "c0e3180d7fe243ac93f24e580793b6ca" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 35519 } }
I20260929 18:41:21.939854 30771 catalog_manager.cc:5495] T e1648bca6d2f45bb9e3b02510e8ed4e6 P c0e3180d7fe243ac93f24e580793b6ca reported cstate change: term changed from 0 to 1, leader changed from <none> to c0e3180d7fe243ac93f24e580793b6ca (127.13.100.3). New cstate: current_term: 1 leader_uuid: "c0e3180d7fe243ac93f24e580793b6ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1582c75505e0403ebfaefca76ccecf35" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 41735 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24b6f50bface46e69c3ee7f7f75a2d49" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 44151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0e3180d7fe243ac93f24e580793b6ca" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 35519 } health_report { overall_health: HEALTHY } } }
I20260929 18:41:21.955493 31248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:21.952093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51864 (local address 127.13.100.3:35519)
0929 18:41:21.952206 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0929 18:41:21.952209 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:41:21.952222 (+ 13us) server_negotiation.cc:407] Connection header received
0929 18:41:21.952356 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:41:21.952358 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:41:21.952414 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:41:21.952499 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:41:21.952754 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:21.953250 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:21.954164 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:21.954367 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:21.954453 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:21.954692 (+ 239us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:41:21.955081 (+ 389us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:41:21.955211 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0929 18:41:21.955276 (+ 65us) server_negotiation.cc:299] Negotiation successful
0929 18:41:21.955329 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 18:41:21.961055 31052 raft_consensus.cc:1261] T e1648bca6d2f45bb9e3b02510e8ed4e6 P 1582c75505e0403ebfaefca76ccecf35 [term 1 FOLLOWER]: Refusing update from remote peer c0e3180d7fe243ac93f24e580793b6ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:41:21.961072 30918 raft_consensus.cc:1261] T e1648bca6d2f45bb9e3b02510e8ed4e6 P 24b6f50bface46e69c3ee7f7f75a2d49 [term 1 FOLLOWER]: Refusing update from remote peer c0e3180d7fe243ac93f24e580793b6ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:41:21.961709 31257 consensus_queue.cc:1035] T e1648bca6d2f45bb9e3b02510e8ed4e6 P c0e3180d7fe243ac93f24e580793b6ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "1582c75505e0403ebfaefca76ccecf35" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:41:21.962010 31259 consensus_queue.cc:1035] T e1648bca6d2f45bb9e3b02510e8ed4e6 P c0e3180d7fe243ac93f24e580793b6ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "24b6f50bface46e69c3ee7f7f75a2d49" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 44151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:41:21.968858 31264 mvcc.cc:205] Tried to move back new op lower bound from 7334737026905264128 to 7334737026816049152. Current Snapshot: MvccSnapshot[applied={T|T < 7334737026905264128}]
I20260929 18:41:21.971405 31263 mvcc.cc:205] Tried to move back new op lower bound from 7334737026905264128 to 7334737026816049152. Current Snapshot: MvccSnapshot[applied={T|T < 7334737026905264128}]
I20260929 18:41:21.976982 30771 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56904:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:41:21.977157 30771 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56904:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:41:21.979558 30771 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3485b2fdf4c340f0b73d535354170a23: Sending DeleteTablet for 3 replicas of tablet e1648bca6d2f45bb9e3b02510e8ed4e6
I20260929 18:41:21.980319 31032 tablet_service.cc:1514] Processing DeleteTablet for tablet e1648bca6d2f45bb9e3b02510e8ed4e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:41:21 UTC) from {username='kudu', principal='kudu/127.13.100.62@KRBTEST.COM'} at 127.0.0.1:38606
I20260929 18:41:21.980370 30898 tablet_service.cc:1514] Processing DeleteTablet for tablet e1648bca6d2f45bb9e3b02510e8ed4e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:41:21 UTC) from {username='kudu', principal='kudu/127.13.100.62@KRBTEST.COM'} at 127.0.0.1:34248
I20260929 18:41:21.980846 31272 tablet_replica.cc:330] T e1648bca6d2f45bb9e3b02510e8ed4e6 P 24b6f50bface46e69c3ee7f7f75a2d49: stopping tablet replica
I20260929 18:41:21.980991 31166 tablet_service.cc:1514] Processing DeleteTablet for tablet e1648bca6d2f45bb9e3b02510e8ed4e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:41:21 UTC) from {username='kudu', principal='kudu/127.13.100.62@KRBTEST.COM'} at 127.0.0.1:51860
I20260929 18:41:21.981197 31271 tablet_replica.cc:330] T e1648bca6d2f45bb9e3b02510e8ed4e6 P 1582c75505e0403ebfaefca76ccecf35: stopping tablet replica
I20260929 18:41:21.981462 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 30835
I20260929 18:41:21.981521 31271 raft_consensus.cc:2229] T e1648bca6d2f45bb9e3b02510e8ed4e6 P 1582c75505e0403ebfaefca76ccecf35 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:41:21.981820 31271 raft_consensus.cc:2258] T e1648bca6d2f45bb9e3b02510e8ed4e6 P 1582c75505e0403ebfaefca76ccecf35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:41:21.984174 31271 ts_tablet_manager.cc:1905] T e1648bca6d2f45bb9e3b02510e8ed4e6 P 1582c75505e0403ebfaefca76ccecf35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:41:21.984350 31273 tablet_replica.cc:330] T e1648bca6d2f45bb9e3b02510e8ed4e6 P c0e3180d7fe243ac93f24e580793b6ca: stopping tablet replica
I20260929 18:41:21.984704 31273 raft_consensus.cc:2229] T e1648bca6d2f45bb9e3b02510e8ed4e6 P c0e3180d7fe243ac93f24e580793b6ca [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:41:21.985163 31273 raft_consensus.cc:2258] T e1648bca6d2f45bb9e3b02510e8ed4e6 P c0e3180d7fe243ac93f24e580793b6ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:41:21.986596 31273 ts_tablet_manager.cc:1905] T e1648bca6d2f45bb9e3b02510e8ed4e6 P c0e3180d7fe243ac93f24e580793b6ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:41:21.987172 31271 ts_tablet_manager.cc:1918] T e1648bca6d2f45bb9e3b02510e8ed4e6 P 1582c75505e0403ebfaefca76ccecf35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:41:21.987318 31271 log.cc:1196] T e1648bca6d2f45bb9e3b02510e8ed4e6 P 1582c75505e0403ebfaefca76ccecf35: Deleting WAL directory at /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707036041078-13712-0/minicluster-data/ts-1/wal/wals/e1648bca6d2f45bb9e3b02510e8ed4e6
I20260929 18:41:21.987713 31271 ts_tablet_manager.cc:1939] T e1648bca6d2f45bb9e3b02510e8ed4e6 P 1582c75505e0403ebfaefca76ccecf35: Deleting consensus metadata
I20260929 18:41:21.988917 30760 catalog_manager.cc:4841] TS 1582c75505e0403ebfaefca76ccecf35 (127.13.100.2:41735): tablet e1648bca6d2f45bb9e3b02510e8ed4e6 (table test-table [id=ea0dc243a33c41da8dd1a56df074a5d3]) successfully deleted
W20260929 18:41:21.989418 30759 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.100.1:44151 (error 108)
I20260929 18:41:21.989807 31273 ts_tablet_manager.cc:1918] T e1648bca6d2f45bb9e3b02510e8ed4e6 P c0e3180d7fe243ac93f24e580793b6ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:41:21.990006 31273 log.cc:1196] T e1648bca6d2f45bb9e3b02510e8ed4e6 P c0e3180d7fe243ac93f24e580793b6ca: Deleting WAL directory at /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707036041078-13712-0/minicluster-data/ts-2/wal/wals/e1648bca6d2f45bb9e3b02510e8ed4e6
I20260929 18:41:21.990247 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 30969
I20260929 18:41:21.990433 31244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:21.990151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.1:44151 (local address 127.0.0.1:34254)
0929 18:41:21.990254 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:21.990338 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.1:44151: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260929 18:41:21.990408 31273 ts_tablet_manager.cc:1939] T e1648bca6d2f45bb9e3b02510e8ed4e6 P c0e3180d7fe243ac93f24e580793b6ca: Deleting consensus metadata
W20260929 18:41:21.994433 30759 catalog_manager.cc:4584] TS 24b6f50bface46e69c3ee7f7f75a2d49 (127.13.100.1:44151): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e1648bca6d2f45bb9e3b02510e8ed4e6: Network error: Client connection negotiation failed: client connection to 127.13.100.1:44151: connect: Connection refused (error 111)
I20260929 18:41:21.994652 30759 catalog_manager.cc:4841] TS c0e3180d7fe243ac93f24e580793b6ca (127.13.100.3:35519): tablet e1648bca6d2f45bb9e3b02510e8ed4e6 (table test-table [id=ea0dc243a33c41da8dd1a56df074a5d3]) successfully deleted
I20260929 18:41:21.996425 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 31103
I20260929 18:41:22.001734 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 30741
2026-09-29T18:41:22Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2251 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-29T18:41:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:41:22Z Disabled control of system clock
I20260929 18:41:22.031570 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707036041078-13712-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707036041078-13712-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707036041078-13712-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.100.62:39723
--webserver_interface=127.13.100.62
--webserver_port=0
--builtin_ntp_servers=127.13.100.20:46515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.100.62:39723
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 18:41:22.145309 31278 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:41:22.145601 31278 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:41:22.145640 31278 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:41:22.150066 31278 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:41:22.150142 31278 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:41:22.150164 31278 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:41:22.150230 31278 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:41:22.150260 31278 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:41:22.155426 31278 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.100.20:46515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707036041078-13712-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707036041078-13712-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.100.62:39723
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.100.62:39723
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.100.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31278
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707036041078-13712-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:41:22.156633 31278 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:41:22.158058 31278 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:41:22.166181 31286 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:41:22.166191 31283 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:41:22.166240 31284 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:41:22.166501 31278 server_base.cc:1034] running on GCE node
I20260929 18:41:22.167100 31278 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:41:22.168172 31278 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:41:22.169363 31278 hybrid_clock.cc:648] HybridClock initialized: now 1790707282169331 us; error 44 us; skew 500 ppm
I20260929 18:41:22.171641 31278 webserver.cc:460] Webserver started at http://127.13.100.62:33955/ using document root <none> and password file <none>
I20260929 18:41:22.172279 31278 fs_manager.cc:359] Metadata directory not provided
I20260929 18:41:22.172375 31278 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:41:22.172616 31278 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:41:22.174748 31278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707036041078-13712-0/minicluster-data/master-0/data/instance:
uuid: "30a3748deec14a3a9527730ef2e9d290"
format_stamp: "Formatted at 2026-09-29 18:41:22 on dist-test-slave-j1xs"
server_key: "336fff7d9bdfd1d3c5101e3b7e6b538a"
server_key_iv: "61b2cfba88a228ae176b2ac0f7dafe08"
server_key_version: "encryptionkey@0"
I20260929 18:41:22.175305 31278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707036041078-13712-0/minicluster-data/master-0/wal/instance:
uuid: "30a3748deec14a3a9527730ef2e9d290"
format_stamp: "Formatted at 2026-09-29 18:41:22 on dist-test-slave-j1xs"
server_key: "336fff7d9bdfd1d3c5101e3b7e6b538a"
server_key_iv: "61b2cfba88a228ae176b2ac0f7dafe08"
server_key_version: "encryptionkey@0"
I20260929 18:41:22.179019 31278 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260929 18:41:22.181785 31292 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:41:22.183115 31278 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260929 18:41:22.183264 31278 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707036041078-13712-0/minicluster-data/master-0/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707036041078-13712-0/minicluster-data/master-0/wal
uuid: "30a3748deec14a3a9527730ef2e9d290"
format_stamp: "Formatted at 2026-09-29 18:41:22 on dist-test-slave-j1xs"
server_key: "336fff7d9bdfd1d3c5101e3b7e6b538a"
server_key_iv: "61b2cfba88a228ae176b2ac0f7dafe08"
server_key_version: "encryptionkey@0"
I20260929 18:41:22.183393 31278 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707036041078-13712-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707036041078-13712-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707036041078-13712-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:41:22.202543 31278 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:41:22.203264 31278 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:41:22.203482 31278 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:41:22.210660 31278 rpc_server.cc:307] RPC server started. Bound to: 127.13.100.62:39723
I20260929 18:41:22.210713 31344 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.100.62:39723 every 8 connection(s)
I20260929 18:41:22.211621 31278 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
I20260929 18:41:22.214783 31345 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:41:22.217572 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 31278
I20260929 18:41:22.217710 13712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707036041078-13712-0/minicluster-data/master-0/wal/instance
I20260929 18:41:22.218055 13712 external_mini_cluster.cc:1428] Setting key 1945d557b1f5fbf9ef3a3411544179a0
I20260929 18:41:22.220762 31345 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30a3748deec14a3a9527730ef2e9d290: Bootstrap starting.
I20260929 18:41:22.223311 31345 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 30a3748deec14a3a9527730ef2e9d290: Neither blocks nor log segments found. Creating new log.
I20260929 18:41:22.224206 31345 log.cc:824] T 00000000000000000000000000000000 P 30a3748deec14a3a9527730ef2e9d290: Log is configured to *not* fsync() on all Append() calls
I20260929 18:41:22.226089 31345 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30a3748deec14a3a9527730ef2e9d290: No bootstrap required, opened a new log
I20260929 18:41:22.227574 31348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:22.219511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50754 (local address 127.13.100.62:39723)
0929 18:41:22.220121 (+ 610us) server_negotiation.cc:206] Beginning negotiation
0929 18:41:22.220137 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 18:41:22.220172 (+ 35us) server_negotiation.cc:407] Connection header received
0929 18:41:22.220774 (+ 602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:41:22.220789 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:41:22.221251 (+ 462us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:41:22.221525 (+ 274us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:41:22.221857 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:22.222879 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:22.224183 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:22.224816 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:22.224999 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:22.225854 (+ 855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:41:22.225860 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:41:22.225871 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:41:22.226363 (+ 492us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:41:22.226428 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:41:22.226448 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:41:22.226555 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 18:41:22.226664 (+ 109us) server_negotiation.cc:299] Negotiation successful
0929 18:41:22.226908 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":131,"threads_started":1}
I20260929 18:41:22.229663 31345 raft_consensus.cc:348] T 00000000000000000000000000000000 P 30a3748deec14a3a9527730ef2e9d290 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30a3748deec14a3a9527730ef2e9d290" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 39723 } }
I20260929 18:41:22.229923 31345 raft_consensus.cc:374] T 00000000000000000000000000000000 P 30a3748deec14a3a9527730ef2e9d290 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:41:22.230007 31345 raft_consensus.cc:729] T 00000000000000000000000000000000 P 30a3748deec14a3a9527730ef2e9d290 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30a3748deec14a3a9527730ef2e9d290, State: Initialized, Role: FOLLOWER
I20260929 18:41:22.230412 31345 consensus_queue.cc:260] T 00000000000000000000000000000000 P 30a3748deec14a3a9527730ef2e9d290 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30a3748deec14a3a9527730ef2e9d290" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 39723 } }
I20260929 18:41:22.230530 31345 raft_consensus.cc:388] T 00000000000000000000000000000000 P 30a3748deec14a3a9527730ef2e9d290 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:41:22.230574 31345 raft_consensus.cc:482] T 00000000000000000000000000000000 P 30a3748deec14a3a9527730ef2e9d290 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:41:22.230643 31345 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 30a3748deec14a3a9527730ef2e9d290 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:41:22.231454 31345 raft_consensus.cc:504] T 00000000000000000000000000000000 P 30a3748deec14a3a9527730ef2e9d290 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30a3748deec14a3a9527730ef2e9d290" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 39723 } }
I20260929 18:41:22.231768 31345 leader_election.cc:302] T 00000000000000000000000000000000 P 30a3748deec14a3a9527730ef2e9d290 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 30a3748deec14a3a9527730ef2e9d290; no voters:
I20260929 18:41:22.232033 31345 leader_election.cc:288] T 00000000000000000000000000000000 P 30a3748deec14a3a9527730ef2e9d290 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:41:22.232173 31350 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 30a3748deec14a3a9527730ef2e9d290 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:41:22.232440 31350 raft_consensus.cc:686] T 00000000000000000000000000000000 P 30a3748deec14a3a9527730ef2e9d290 [term 1 LEADER]: Becoming Leader. State: Replica: 30a3748deec14a3a9527730ef2e9d290, State: Running, Role: LEADER
I20260929 18:41:22.232836 31350 consensus_queue.cc:237] T 00000000000000000000000000000000 P 30a3748deec14a3a9527730ef2e9d290 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30a3748deec14a3a9527730ef2e9d290" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 39723 } }
I20260929 18:41:22.233263 31345 sys_catalog.cc:564] T 00000000000000000000000000000000 P 30a3748deec14a3a9527730ef2e9d290 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:41:22.234722 31351 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30a3748deec14a3a9527730ef2e9d290 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "30a3748deec14a3a9527730ef2e9d290" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30a3748deec14a3a9527730ef2e9d290" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 39723 } } }
I20260929 18:41:22.234783 31352 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30a3748deec14a3a9527730ef2e9d290 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 30a3748deec14a3a9527730ef2e9d290. Latest consensus state: current_term: 1 leader_uuid: "30a3748deec14a3a9527730ef2e9d290" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30a3748deec14a3a9527730ef2e9d290" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 39723 } } }
I20260929 18:41:22.234848 31351 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30a3748deec14a3a9527730ef2e9d290 [sys.catalog]: This master's current role is: LEADER
I20260929 18:41:22.234891 31352 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30a3748deec14a3a9527730ef2e9d290 [sys.catalog]: This master's current role is: LEADER
I20260929 18:41:22.235216 31359 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:41:22.238294 31359 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:41:22.244022 31359 catalog_manager.cc:1347] Generated new cluster ID: f2b08dd0977b4c1181bfa61e39024c61
I20260929 18:41:22.244112 31359 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:41:22.270730 31359 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:41:22.271694 31359 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:41:22.277240 31359 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 30a3748deec14a3a9527730ef2e9d290: Generated new TSK 0
I20260929 18:41:22.277974 31359 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:41:22.291795 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707036041078-13712-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707036041078-13712-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707036041078-13712-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707036041078-13712-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.100.1:0
--local_ip_for_outbound_sockets=127.13.100.1
--webserver_interface=127.13.100.1
--webserver_port=0
--tserver_master_addrs=127.13.100.62:39723
--builtin_ntp_servers=127.13.100.20:46515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 18:41:22.399894 31369 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:41:22.400159 31369 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:41:22.400198 31369 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:41:22.404186 31369 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:41:22.404243 31369 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:41:22.404331 31369 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.100.1
I20260929 18:41:22.408978 31369 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.100.20:46515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707036041078-13712-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707036041078-13712-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.100.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707036041078-13712-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.100.1
--webserver_port=0
--tserver_master_addrs=127.13.100.62:39723
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31369
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.100.1
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707036041078-13712-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:41:22.410004 31369 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:41:22.411393 31369 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:41:22.421169 31377 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:41:22.421134 31374 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:41:22.421268 31369 server_base.cc:1034] running on GCE node
W20260929 18:41:22.421134 31375 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:41:22.421741 31369 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:41:22.422387 31369 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:41:22.423550 31369 hybrid_clock.cc:648] HybridClock initialized: now 1790707282423518 us; error 46 us; skew 500 ppm
I20260929 18:41:22.425374 31369 webserver.cc:460] Webserver started at http://127.13.100.1:46535/ using document root <none> and password file <none>
I20260929 18:41:22.426007 31369 fs_manager.cc:359] Metadata directory not provided
I20260929 18:41:22.426075 31369 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:41:22.426282 31369 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:41:22.428287 31369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707036041078-13712-0/minicluster-data/ts-0/data/instance:
uuid: "4830904637f8434bbf706ad9092d4e1b"
format_stamp: "Formatted at 2026-09-29 18:41:22 on dist-test-slave-j1xs"
server_key: "1337fac748bd6f05875ed410b4b107da"
server_key_iv: "1a3f58e06e0acfd5fdb28b27990746b8"
server_key_version: "encryptionkey@0"
I20260929 18:41:22.428826 31369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707036041078-13712-0/minicluster-data/ts-0/wal/instance:
uuid: "4830904637f8434bbf706ad9092d4e1b"
format_stamp: "Formatted at 2026-09-29 18:41:22 on dist-test-slave-j1xs"
server_key: "1337fac748bd6f05875ed410b4b107da"
server_key_iv: "1a3f58e06e0acfd5fdb28b27990746b8"
server_key_version: "encryptionkey@0"
I20260929 18:41:22.432327 31369 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260929 18:41:22.434790 31383 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:41:22.436079 31369 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:41:22.436204 31369 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707036041078-13712-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707036041078-13712-0/minicluster-data/ts-0/wal
uuid: "4830904637f8434bbf706ad9092d4e1b"
format_stamp: "Formatted at 2026-09-29 18:41:22 on dist-test-slave-j1xs"
server_key: "1337fac748bd6f05875ed410b4b107da"
server_key_iv: "1a3f58e06e0acfd5fdb28b27990746b8"
server_key_version: "encryptionkey@0"
I20260929 18:41:22.436321 31369 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707036041078-13712-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707036041078-13712-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707036041078-13712-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:41:22.448527 31369 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:41:22.449208 31369 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:41:22.449409 31369 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:41:22.450078 31369 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:41:22.451112 31369 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:41:22.451179 31369 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:41:22.451246 31369 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:41:22.451297 31369 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:41:22.460616 31369 rpc_server.cc:307] RPC server started. Bound to: 127.13.100.1:39575
I20260929 18:41:22.460666 31496 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.100.1:39575 every 8 connection(s)
I20260929 18:41:22.461629 31369 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707036041078-13712-0/minicluster-data/ts-0/data/info.pb
I20260929 18:41:22.468185 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 31369
I20260929 18:41:22.468302 13712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707036041078-13712-0/minicluster-data/ts-0/wal/instance
I20260929 18:41:22.468590 13712 external_mini_cluster.cc:1428] Setting key 391dd0ed6297452fad74fe3a9e9b2df0
I20260929 18:41:22.470839 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707036041078-13712-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707036041078-13712-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707036041078-13712-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707036041078-13712-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.100.2:0
--local_ip_for_outbound_sockets=127.13.100.2
--webserver_interface=127.13.100.2
--webserver_port=0
--tserver_master_addrs=127.13.100.62:39723
--builtin_ntp_servers=127.13.100.20:46515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 18:41:22.472478 31348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:22.463621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.1:47457 (local address 127.13.100.62:39723)
0929 18:41:22.463798 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0929 18:41:22.463803 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:41:22.464866 (+ 1063us) server_negotiation.cc:407] Connection header received
0929 18:41:22.466225 (+ 1359us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:41:22.466228 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:41:22.466274 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:41:22.466351 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:41:22.467260 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:22.467983 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:22.470004 (+ 2021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:22.470303 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:22.470392 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:22.471823 (+ 1431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:41:22.471826 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:41:22.471828 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:41:22.471841 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:41:22.471863 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:41:22.471871 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:41:22.471937 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0929 18:41:22.472165 (+ 228us) server_negotiation.cc:299] Negotiation successful
0929 18:41:22.472260 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 18:41:22.473387 31499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:22.464032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:39723 (local address 127.13.100.1:47457)
0929 18:41:22.464708 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:22.464749 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 18:41:22.466022 (+ 1273us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:41:22.466538 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:41:22.466544 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:41:22.466626 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:41:22.467113 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:41:22.467125 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:41:22.468153 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:41:22.468156 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:41:22.469639 (+ 1483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:41:22.469650 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:41:22.470480 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:41:22.470484 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:41:22.470731 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:22.471468 (+ 737us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:41:22.471472 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:41:22.471539 (+ 67us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:41:22.471640 (+ 101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:41:22.471961 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:41:22.471972 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:41:22.471982 (+ 10us) client_negotiation.cc:769] Sending connection context
0929 18:41:22.472157 (+ 175us) client_negotiation.cc:240] Negotiation successful
0929 18:41:22.472519 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":128,"threads_started":1}
I20260929 18:41:22.474752 31497 heartbeater.cc:347] Connected to a master server at 127.13.100.62:39723
I20260929 18:41:22.475031 31497 heartbeater.cc:464] Registering TS with master...
I20260929 18:41:22.475556 31497 heartbeater.cc:511] Master 127.13.100.62:39723 requested a full tablet report, sending...
I20260929 18:41:22.477286 31309 ts_manager.cc:194] Registered new tserver with Master: 4830904637f8434bbf706ad9092d4e1b (127.13.100.1:39575)
I20260929 18:41:22.479079 31309 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.100.1:47457
W20260929 18:41:22.584897 31500 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:41:22.585177 31500 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:41:22.585218 31500 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:41:22.589540 31500 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:41:22.589609 31500 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:41:22.589668 31500 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.100.2
I20260929 18:41:22.594616 31500 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.100.20:46515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707036041078-13712-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707036041078-13712-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.100.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707036041078-13712-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.100.2
--webserver_port=0
--tserver_master_addrs=127.13.100.62:39723
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31500
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.100.2
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707036041078-13712-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:41:22.595685 31500 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:41:22.597069 31500 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:41:22.606886 31505 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:41:22.606860 31506 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:41:22.606889 31508 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:41:22.607270 31500 server_base.cc:1034] running on GCE node
I20260929 18:41:22.607647 31500 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:41:22.608245 31500 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:41:22.609421 31500 hybrid_clock.cc:648] HybridClock initialized: now 1790707282609406 us; error 28 us; skew 500 ppm
I20260929 18:41:22.611435 31500 webserver.cc:460] Webserver started at http://127.13.100.2:45705/ using document root <none> and password file <none>
I20260929 18:41:22.611975 31500 fs_manager.cc:359] Metadata directory not provided
I20260929 18:41:22.612037 31500 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:41:22.612195 31500 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:41:22.614290 31500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707036041078-13712-0/minicluster-data/ts-1/data/instance:
uuid: "82929379546a4715b81ee33bd65b5d73"
format_stamp: "Formatted at 2026-09-29 18:41:22 on dist-test-slave-j1xs"
server_key: "6b7979b04e6a1febf742d32af1489d51"
server_key_iv: "972e0bd99773b6771c6f0298ff2909d2"
server_key_version: "encryptionkey@0"
I20260929 18:41:22.614801 31500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707036041078-13712-0/minicluster-data/ts-1/wal/instance:
uuid: "82929379546a4715b81ee33bd65b5d73"
format_stamp: "Formatted at 2026-09-29 18:41:22 on dist-test-slave-j1xs"
server_key: "6b7979b04e6a1febf742d32af1489d51"
server_key_iv: "972e0bd99773b6771c6f0298ff2909d2"
server_key_version: "encryptionkey@0"
I20260929 18:41:22.618445 31500 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260929 18:41:22.621110 31514 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:41:22.622777 31500 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:41:22.622937 31500 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707036041078-13712-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707036041078-13712-0/minicluster-data/ts-1/wal
uuid: "82929379546a4715b81ee33bd65b5d73"
format_stamp: "Formatted at 2026-09-29 18:41:22 on dist-test-slave-j1xs"
server_key: "6b7979b04e6a1febf742d32af1489d51"
server_key_iv: "972e0bd99773b6771c6f0298ff2909d2"
server_key_version: "encryptionkey@0"
I20260929 18:41:22.623061 31500 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707036041078-13712-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707036041078-13712-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707036041078-13712-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:41:22.633277 31500 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:41:22.634017 31500 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:41:22.634244 31500 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:41:22.634814 31500 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:41:22.635818 31500 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:41:22.635885 31500 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:41:22.635955 31500 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:41:22.636008 31500 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:41:22.645044 31500 rpc_server.cc:307] RPC server started. Bound to: 127.13.100.2:33939
I20260929 18:41:22.645066 31627 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.100.2:33939 every 8 connection(s)
I20260929 18:41:22.646008 31500 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707036041078-13712-0/minicluster-data/ts-1/data/info.pb
I20260929 18:41:22.647928 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 31500
I20260929 18:41:22.648059 13712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707036041078-13712-0/minicluster-data/ts-1/wal/instance
I20260929 18:41:22.648360 13712 external_mini_cluster.cc:1428] Setting key 4153539a644035c1dd68f900db62b77b
I20260929 18:41:22.650345 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707036041078-13712-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707036041078-13712-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707036041078-13712-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707036041078-13712-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.100.3:0
--local_ip_for_outbound_sockets=127.13.100.3
--webserver_interface=127.13.100.3
--webserver_port=0
--tserver_master_addrs=127.13.100.62:39723
--builtin_ntp_servers=127.13.100.20:46515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 18:41:22.656739 31348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:22.647969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.2:60073 (local address 127.13.100.62:39723)
0929 18:41:22.648136 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0929 18:41:22.648142 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:41:22.649334 (+ 1192us) server_negotiation.cc:407] Connection header received
0929 18:41:22.650873 (+ 1539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:41:22.650879 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:41:22.650948 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:41:22.651040 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:41:22.652219 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:22.652741 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:22.654577 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:22.654819 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:22.654913 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:22.656267 (+ 1354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:41:22.656271 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:41:22.656273 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:41:22.656286 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:41:22.656304 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:41:22.656312 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:41:22.656370 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0929 18:41:22.656570 (+ 200us) server_negotiation.cc:299] Negotiation successful
0929 18:41:22.656623 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 18:41:22.658082 31630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:22.648461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:39723 (local address 127.13.100.2:60073)
0929 18:41:22.649187 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:22.649238 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 18:41:22.650660 (+ 1422us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:41:22.651207 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:41:22.651214 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:41:22.651328 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:41:22.652056 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:41:22.652077 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:41:22.652881 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:41:22.652887 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:41:22.654430 (+ 1543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:41:22.654444 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:41:22.654995 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:41:22.655002 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:41:22.655253 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:22.655946 (+ 693us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:41:22.655949 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:41:22.656011 (+ 62us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:41:22.656106 (+ 95us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:41:22.656401 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:41:22.656412 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:41:22.656422 (+ 10us) client_negotiation.cc:769] Sending connection context
0929 18:41:22.656570 (+ 148us) client_negotiation.cc:240] Negotiation successful
0929 18:41:22.656919 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":135,"threads_started":1}
I20260929 18:41:22.659382 31628 heartbeater.cc:347] Connected to a master server at 127.13.100.62:39723
I20260929 18:41:22.659654 31628 heartbeater.cc:464] Registering TS with master...
I20260929 18:41:22.660202 31628 heartbeater.cc:511] Master 127.13.100.62:39723 requested a full tablet report, sending...
I20260929 18:41:22.661389 31309 ts_manager.cc:194] Registered new tserver with Master: 82929379546a4715b81ee33bd65b5d73 (127.13.100.2:33939)
I20260929 18:41:22.662076 31309 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.100.2:60073
W20260929 18:41:22.763173 31631 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:41:22.763517 31631 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:41:22.763578 31631 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:41:22.768193 31631 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:41:22.768246 31631 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:41:22.768303 31631 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.100.3
I20260929 18:41:22.773053 31631 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.100.20:46515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707036041078-13712-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707036041078-13712-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.100.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707036041078-13712-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.100.3
--webserver_port=0
--tserver_master_addrs=127.13.100.62:39723
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31631
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.100.3
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707036041078-13712-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:41:22.774294 31631 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:41:22.775722 31631 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 18:41:22.785442 31631 server_base.cc:1034] running on GCE node
W20260929 18:41:22.785355 31637 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:41:22.785328 31639 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:41:22.785343 31636 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:41:22.786041 31631 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:41:22.786752 31631 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:41:22.787942 31631 hybrid_clock.cc:648] HybridClock initialized: now 1790707282787925 us; error 30 us; skew 500 ppm
I20260929 18:41:22.789726 31631 webserver.cc:460] Webserver started at http://127.13.100.3:36833/ using document root <none> and password file <none>
I20260929 18:41:22.790300 31631 fs_manager.cc:359] Metadata directory not provided
I20260929 18:41:22.790359 31631 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:41:22.790583 31631 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:41:22.792518 31631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707036041078-13712-0/minicluster-data/ts-2/data/instance:
uuid: "0101fca067db410cb59233fd2436b9c2"
format_stamp: "Formatted at 2026-09-29 18:41:22 on dist-test-slave-j1xs"
server_key: "984fc52f8638ff7d05e7190a8478e076"
server_key_iv: "0dd04793073673c80528d8e02ff7f040"
server_key_version: "encryptionkey@0"
I20260929 18:41:22.793073 31631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707036041078-13712-0/minicluster-data/ts-2/wal/instance:
uuid: "0101fca067db410cb59233fd2436b9c2"
format_stamp: "Formatted at 2026-09-29 18:41:22 on dist-test-slave-j1xs"
server_key: "984fc52f8638ff7d05e7190a8478e076"
server_key_iv: "0dd04793073673c80528d8e02ff7f040"
server_key_version: "encryptionkey@0"
I20260929 18:41:22.796962 31631 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 18:41:22.799475 31645 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:41:22.800806 31631 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 18:41:22.800957 31631 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707036041078-13712-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707036041078-13712-0/minicluster-data/ts-2/wal
uuid: "0101fca067db410cb59233fd2436b9c2"
format_stamp: "Formatted at 2026-09-29 18:41:22 on dist-test-slave-j1xs"
server_key: "984fc52f8638ff7d05e7190a8478e076"
server_key_iv: "0dd04793073673c80528d8e02ff7f040"
server_key_version: "encryptionkey@0"
I20260929 18:41:22.801080 31631 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707036041078-13712-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707036041078-13712-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707036041078-13712-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:41:22.831681 31631 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:41:22.832386 31631 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:41:22.832602 31631 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:41:22.833204 31631 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:41:22.834247 31631 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:41:22.834321 31631 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:41:22.834391 31631 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:41:22.834455 31631 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:41:22.843667 31631 rpc_server.cc:307] RPC server started. Bound to: 127.13.100.3:46597
I20260929 18:41:22.843721 31758 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.100.3:46597 every 8 connection(s)
I20260929 18:41:22.844591 31631 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707036041078-13712-0/minicluster-data/ts-2/data/info.pb
I20260929 18:41:22.847668 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 31631
I20260929 18:41:22.847791 13712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707036041078-13712-0/minicluster-data/ts-2/wal/instance
I20260929 18:41:22.848109 13712 external_mini_cluster.cc:1428] Setting key b265ef05ac12d5572fcd3320ae52ca5c
I20260929 18:41:22.854112 31348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:22.846505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.3:33169 (local address 127.13.100.62:39723)
0929 18:41:22.846677 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0929 18:41:22.846682 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:41:22.847781 (+ 1099us) server_negotiation.cc:407] Connection header received
0929 18:41:22.849450 (+ 1669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:41:22.849453 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:41:22.849517 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:41:22.849589 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:41:22.850625 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:22.851157 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:22.852345 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:22.852573 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:22.852664 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:22.853645 (+ 981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:41:22.853648 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:41:22.853651 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:41:22.853662 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:41:22.853686 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:41:22.853698 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:41:22.853771 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 18:41:22.853941 (+ 170us) server_negotiation.cc:299] Negotiation successful
0929 18:41:22.854003 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 18:41:22.854930 31761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:22.846848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:39723 (local address 127.13.100.3:33169)
0929 18:41:22.847555 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:22.847609 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 18:41:22.849195 (+ 1586us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:41:22.849765 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:41:22.849771 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:41:22.849849 (+ 78us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:41:22.850497 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:41:22.850512 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:41:22.851263 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:41:22.851265 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:41:22.852234 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:41:22.852243 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:41:22.852702 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:41:22.852704 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:41:22.852830 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:22.853402 (+ 572us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:41:22.853404 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:41:22.853447 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:41:22.853503 (+ 56us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:41:22.853788 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:41:22.853794 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:41:22.853798 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 18:41:22.853949 (+ 151us) client_negotiation.cc:240] Negotiation successful
0929 18:41:22.854162 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":109,"threads_started":1}
I20260929 18:41:22.855978 31759 heartbeater.cc:347] Connected to a master server at 127.13.100.62:39723
I20260929 18:41:22.856225 31759 heartbeater.cc:464] Registering TS with master...
I20260929 18:41:22.856709 31759 heartbeater.cc:511] Master 127.13.100.62:39723 requested a full tablet report, sending...
I20260929 18:41:22.857682 31309 ts_manager.cc:194] Registered new tserver with Master: 0101fca067db410cb59233fd2436b9c2 (127.13.100.3:46597)
I20260929 18:41:22.858291 31309 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.100.3:33169
I20260929 18:41:22.862021 13712 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 18:41:22.867578 31348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:22.863931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50756 (local address 127.13.100.62:39723)
0929 18:41:22.864028 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0929 18:41:22.864032 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:41:22.864295 (+ 263us) server_negotiation.cc:407] Connection header received
0929 18:41:22.864359 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:41:22.864361 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:41:22.864401 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:41:22.864480 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:41:22.864789 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:22.865288 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:22.866163 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:22.866368 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:22.866456 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:22.867241 (+ 785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:41:22.867244 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:41:22.867246 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:41:22.867256 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:41:22.867272 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:41:22.867279 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:41:22.867332 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0929 18:41:22.867430 (+ 98us) server_negotiation.cc:299] Negotiation successful
0929 18:41:22.867477 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260929 18:41:22.870297 31309 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50756:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:41:22.873216 31309 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:41:22.887534 31774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:22.882189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.2:33939 (local address 127.0.0.1:51374)
0929 18:41:22.882967 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:22.882979 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:41:22.883070 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:41:22.883546 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:41:22.883549 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:41:22.883564 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:41:22.883637 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:41:22.883642 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:41:22.884758 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:41:22.884764 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:41:22.886055 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:41:22.886066 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:41:22.886994 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:41:22.886999 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:41:22.887103 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:22.887156 (+ 53us) client_negotiation.cc:769] Sending connection context
0929 18:41:22.887256 (+ 100us) client_negotiation.cc:240] Negotiation successful
0929 18:41:22.887370 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":721,"thread_start_us":91,"threads_started":1}
I20260929 18:41:22.887692 31772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:22.881496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36236 (local address 127.13.100.3:46597)
0929 18:41:22.882175 (+ 679us) server_negotiation.cc:206] Beginning negotiation
0929 18:41:22.882187 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 18:41:22.882510 (+ 323us) server_negotiation.cc:407] Connection header received
0929 18:41:22.882591 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:41:22.882596 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:41:22.882824 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:41:22.883015 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:41:22.883537 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:22.884302 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:22.886207 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:22.887104 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:22.887205 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:22.887379 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0929 18:41:22.887428 (+ 49us) server_negotiation.cc:299] Negotiation successful
0929 18:41:22.887515 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":566,"thread_start_us":107,"threads_started":1}
I20260929 18:41:22.888005 31775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:22.882603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51374 (local address 127.13.100.2:33939)
0929 18:41:22.882949 (+ 346us) server_negotiation.cc:206] Beginning negotiation
0929 18:41:22.882959 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 18:41:22.883048 (+ 89us) server_negotiation.cc:407] Connection header received
0929 18:41:22.883187 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:41:22.883191 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:41:22.883377 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:41:22.883539 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:41:22.883722 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:22.884513 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:22.886196 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:22.886785 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:22.886883 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:22.887068 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0929 18:41:22.887758 (+ 690us) server_negotiation.cc:299] Negotiation successful
0929 18:41:22.887861 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":117,"threads_started":1}
I20260929 18:41:22.888528 31770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:22.881495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.1:39575 (local address 127.0.0.1:35208)
0929 18:41:22.882316 (+ 821us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:22.882329 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:41:22.882416 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:41:22.883243 (+ 827us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:41:22.883245 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:41:22.883262 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:41:22.883418 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:41:22.883424 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:41:22.885683 (+ 2259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:41:22.885686 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:41:22.886702 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:41:22.886712 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:41:22.888149 (+ 1437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:41:22.888152 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:41:22.888249 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:22.888271 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 18:41:22.888341 (+ 70us) client_negotiation.cc:240] Negotiation successful
0929 18:41:22.888403 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":744,"mutex_wait_us":3,"thread_start_us":95,"threads_started":1}
I20260929 18:41:22.887653 31771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:22.881400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.3:46597 (local address 127.0.0.1:36236)
0929 18:41:22.882198 (+ 798us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:22.882246 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 18:41:22.882364 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:41:22.883184 (+ 820us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:41:22.883190 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:41:22.883212 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:41:22.883407 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:41:22.883414 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:41:22.884427 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:41:22.884430 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:41:22.886093 (+ 1663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:41:22.886103 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:41:22.887207 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:41:22.887211 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:41:22.887298 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:22.887320 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 18:41:22.887382 (+ 62us) client_negotiation.cc:240] Negotiation successful
0929 18:41:22.887455 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":616,"mutex_wait_us":208,"thread_start_us":62,"threads_started":1}
I20260929 18:41:22.889483 31773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:22.881700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35208 (local address 127.13.100.1:39575)
0929 18:41:22.882750 (+ 1050us) server_negotiation.cc:206] Beginning negotiation
0929 18:41:22.882761 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 18:41:22.882786 (+ 25us) server_negotiation.cc:407] Connection header received
0929 18:41:22.882858 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:41:22.882865 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:41:22.883065 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:41:22.883262 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:41:22.884421 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:22.885569 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:22.886829 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:22.887966 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:22.888092 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:22.889196 (+ 1104us) server_negotiation.cc:1035] Waiting for connection context
0929 18:41:22.889242 (+ 46us) server_negotiation.cc:299] Negotiation successful
0929 18:41:22.889327 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":939,"thread_start_us":130,"threads_started":1}
I20260929 18:41:22.891168 31693 tablet_service.cc:1467] Processing CreateTablet for tablet c5313922f223418ea6cbf4670e69a13a (DEFAULT_TABLE table=test-table [id=d536ae43709443d788edfd00fd1d8b89]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:41:22.891197 31562 tablet_service.cc:1467] Processing CreateTablet for tablet c5313922f223418ea6cbf4670e69a13a (DEFAULT_TABLE table=test-table [id=d536ae43709443d788edfd00fd1d8b89]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:41:22.892122 31693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5313922f223418ea6cbf4670e69a13a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:41:22.892155 31562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5313922f223418ea6cbf4670e69a13a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:41:22.891909 31431 tablet_service.cc:1467] Processing CreateTablet for tablet c5313922f223418ea6cbf4670e69a13a (DEFAULT_TABLE table=test-table [id=d536ae43709443d788edfd00fd1d8b89]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:41:22.892791 31431 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5313922f223418ea6cbf4670e69a13a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:41:22.898092 31776 tablet_bootstrap.cc:492] T c5313922f223418ea6cbf4670e69a13a P 0101fca067db410cb59233fd2436b9c2: Bootstrap starting.
I20260929 18:41:22.899581 31778 tablet_bootstrap.cc:492] T c5313922f223418ea6cbf4670e69a13a P 4830904637f8434bbf706ad9092d4e1b: Bootstrap starting.
I20260929 18:41:22.900370 31776 tablet_bootstrap.cc:654] T c5313922f223418ea6cbf4670e69a13a P 0101fca067db410cb59233fd2436b9c2: Neither blocks nor log segments found. Creating new log.
I20260929 18:41:22.901329 31776 log.cc:824] T c5313922f223418ea6cbf4670e69a13a P 0101fca067db410cb59233fd2436b9c2: Log is configured to *not* fsync() on all Append() calls
I20260929 18:41:22.901552 31778 tablet_bootstrap.cc:654] T c5313922f223418ea6cbf4670e69a13a P 4830904637f8434bbf706ad9092d4e1b: Neither blocks nor log segments found. Creating new log.
I20260929 18:41:22.902215 31777 tablet_bootstrap.cc:492] T c5313922f223418ea6cbf4670e69a13a P 82929379546a4715b81ee33bd65b5d73: Bootstrap starting.
I20260929 18:41:22.902500 31778 log.cc:824] T c5313922f223418ea6cbf4670e69a13a P 4830904637f8434bbf706ad9092d4e1b: Log is configured to *not* fsync() on all Append() calls
I20260929 18:41:22.903513 31776 tablet_bootstrap.cc:492] T c5313922f223418ea6cbf4670e69a13a P 0101fca067db410cb59233fd2436b9c2: No bootstrap required, opened a new log
I20260929 18:41:22.903697 31776 ts_tablet_manager.cc:1397] T c5313922f223418ea6cbf4670e69a13a P 0101fca067db410cb59233fd2436b9c2: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 18:41:22.904266 31777 tablet_bootstrap.cc:654] T c5313922f223418ea6cbf4670e69a13a P 82929379546a4715b81ee33bd65b5d73: Neither blocks nor log segments found. Creating new log.
I20260929 18:41:22.904479 31778 tablet_bootstrap.cc:492] T c5313922f223418ea6cbf4670e69a13a P 4830904637f8434bbf706ad9092d4e1b: No bootstrap required, opened a new log
I20260929 18:41:22.904680 31778 ts_tablet_manager.cc:1397] T c5313922f223418ea6cbf4670e69a13a P 4830904637f8434bbf706ad9092d4e1b: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260929 18:41:22.905049 31777 log.cc:824] T c5313922f223418ea6cbf4670e69a13a P 82929379546a4715b81ee33bd65b5d73: Log is configured to *not* fsync() on all Append() calls
I20260929 18:41:22.906687 31777 tablet_bootstrap.cc:492] T c5313922f223418ea6cbf4670e69a13a P 82929379546a4715b81ee33bd65b5d73: No bootstrap required, opened a new log
I20260929 18:41:22.906879 31777 ts_tablet_manager.cc:1397] T c5313922f223418ea6cbf4670e69a13a P 82929379546a4715b81ee33bd65b5d73: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 18:41:22.908077 31776 raft_consensus.cc:348] T c5313922f223418ea6cbf4670e69a13a P 0101fca067db410cb59233fd2436b9c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0101fca067db410cb59233fd2436b9c2" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 46597 } } peers { permanent_uuid: "4830904637f8434bbf706ad9092d4e1b" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 39575 } } peers { permanent_uuid: "82929379546a4715b81ee33bd65b5d73" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 33939 } }
I20260929 18:41:22.908342 31776 raft_consensus.cc:374] T c5313922f223418ea6cbf4670e69a13a P 0101fca067db410cb59233fd2436b9c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:41:22.908397 31776 raft_consensus.cc:729] T c5313922f223418ea6cbf4670e69a13a P 0101fca067db410cb59233fd2436b9c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0101fca067db410cb59233fd2436b9c2, State: Initialized, Role: FOLLOWER
I20260929 18:41:22.908825 31776 consensus_queue.cc:260] T c5313922f223418ea6cbf4670e69a13a P 0101fca067db410cb59233fd2436b9c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0101fca067db410cb59233fd2436b9c2" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 46597 } } peers { permanent_uuid: "4830904637f8434bbf706ad9092d4e1b" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 39575 } } peers { permanent_uuid: "82929379546a4715b81ee33bd65b5d73" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 33939 } }
I20260929 18:41:22.909469 31759 heartbeater.cc:503] Master 127.13.100.62:39723 was elected leader, sending a full tablet report...
I20260929 18:41:22.909833 31776 ts_tablet_manager.cc:1428] T c5313922f223418ea6cbf4670e69a13a P 0101fca067db410cb59233fd2436b9c2: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 18:41:22.908782 31778 raft_consensus.cc:348] T c5313922f223418ea6cbf4670e69a13a P 4830904637f8434bbf706ad9092d4e1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0101fca067db410cb59233fd2436b9c2" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 46597 } } peers { permanent_uuid: "4830904637f8434bbf706ad9092d4e1b" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 39575 } } peers { permanent_uuid: "82929379546a4715b81ee33bd65b5d73" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 33939 } }
I20260929 18:41:22.909953 31778 raft_consensus.cc:374] T c5313922f223418ea6cbf4670e69a13a P 4830904637f8434bbf706ad9092d4e1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:41:22.910012 31778 raft_consensus.cc:729] T c5313922f223418ea6cbf4670e69a13a P 4830904637f8434bbf706ad9092d4e1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4830904637f8434bbf706ad9092d4e1b, State: Initialized, Role: FOLLOWER
I20260929 18:41:22.910460 31778 consensus_queue.cc:260] T c5313922f223418ea6cbf4670e69a13a P 4830904637f8434bbf706ad9092d4e1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0101fca067db410cb59233fd2436b9c2" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 46597 } } peers { permanent_uuid: "4830904637f8434bbf706ad9092d4e1b" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 39575 } } peers { permanent_uuid: "82929379546a4715b81ee33bd65b5d73" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 33939 } }
I20260929 18:41:22.910449 31777 raft_consensus.cc:348] T c5313922f223418ea6cbf4670e69a13a P 82929379546a4715b81ee33bd65b5d73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0101fca067db410cb59233fd2436b9c2" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 46597 } } peers { permanent_uuid: "4830904637f8434bbf706ad9092d4e1b" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 39575 } } peers { permanent_uuid: "82929379546a4715b81ee33bd65b5d73" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 33939 } }
I20260929 18:41:22.910701 31777 raft_consensus.cc:374] T c5313922f223418ea6cbf4670e69a13a P 82929379546a4715b81ee33bd65b5d73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:41:22.910768 31777 raft_consensus.cc:729] T c5313922f223418ea6cbf4670e69a13a P 82929379546a4715b81ee33bd65b5d73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82929379546a4715b81ee33bd65b5d73, State: Initialized, Role: FOLLOWER
I20260929 18:41:22.911145 31497 heartbeater.cc:503] Master 127.13.100.62:39723 was elected leader, sending a full tablet report...
I20260929 18:41:22.911372 31778 ts_tablet_manager.cc:1428] T c5313922f223418ea6cbf4670e69a13a P 4830904637f8434bbf706ad9092d4e1b: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260929 18:41:22.912135 31777 consensus_queue.cc:260] T c5313922f223418ea6cbf4670e69a13a P 82929379546a4715b81ee33bd65b5d73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0101fca067db410cb59233fd2436b9c2" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 46597 } } peers { permanent_uuid: "4830904637f8434bbf706ad9092d4e1b" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 39575 } } peers { permanent_uuid: "82929379546a4715b81ee33bd65b5d73" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 33939 } }
I20260929 18:41:22.912902 31628 heartbeater.cc:503] Master 127.13.100.62:39723 was elected leader, sending a full tablet report...
I20260929 18:41:22.913021 31777 ts_tablet_manager.cc:1428] T c5313922f223418ea6cbf4670e69a13a P 82929379546a4715b81ee33bd65b5d73: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260929 18:41:22.962901 31498 tablet.cc:2369] T c5313922f223418ea6cbf4670e69a13a P 4830904637f8434bbf706ad9092d4e1b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 18:41:23.095577 31760 tablet.cc:2369] T c5313922f223418ea6cbf4670e69a13a P 0101fca067db410cb59233fd2436b9c2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 18:41:23.147118 31629 tablet.cc:2369] T c5313922f223418ea6cbf4670e69a13a P 82929379546a4715b81ee33bd65b5d73: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:41:23.158746 31782 raft_consensus.cc:482] T c5313922f223418ea6cbf4670e69a13a P 0101fca067db410cb59233fd2436b9c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:41:23.159000 31782 raft_consensus.cc:504] T c5313922f223418ea6cbf4670e69a13a P 0101fca067db410cb59233fd2436b9c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0101fca067db410cb59233fd2436b9c2" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 46597 } } peers { permanent_uuid: "4830904637f8434bbf706ad9092d4e1b" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 39575 } } peers { permanent_uuid: "82929379546a4715b81ee33bd65b5d73" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 33939 } }
I20260929 18:41:23.160148 31782 leader_election.cc:288] T c5313922f223418ea6cbf4670e69a13a P 0101fca067db410cb59233fd2436b9c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4830904637f8434bbf706ad9092d4e1b (127.13.100.1:39575), 82929379546a4715b81ee33bd65b5d73 (127.13.100.2:33939)
I20260929 18:41:23.164793 31761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:23.160455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.2:33939 (local address 127.13.100.3:34671)
0929 18:41:23.160666 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:23.160683 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 18:41:23.160805 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:41:23.161241 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:41:23.161244 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:41:23.161283 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:41:23.161524 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:41:23.161529 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:41:23.162611 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:41:23.162615 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:41:23.163656 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:41:23.163666 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:41:23.164389 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:41:23.164392 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:41:23.164487 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:23.164513 (+ 26us) client_negotiation.cc:769] Sending connection context
0929 18:41:23.164584 (+ 71us) client_negotiation.cc:240] Negotiation successful
0929 18:41:23.164639 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"mutex_wait_us":100,"spinlock_wait_cycles":7296}
I20260929 18:41:23.164824 31775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:23.160557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.3:34671 (local address 127.13.100.2:33939)
0929 18:41:23.160698 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0929 18:41:23.160702 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:41:23.160748 (+ 46us) server_negotiation.cc:407] Connection header received
0929 18:41:23.161053 (+ 305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:41:23.161056 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:41:23.161112 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:41:23.161218 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:41:23.161636 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:23.162492 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:23.163773 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:23.164268 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:23.164395 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:23.164446 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0929 18:41:23.164615 (+ 169us) server_negotiation.cc:299] Negotiation successful
0929 18:41:23.164685 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 18:41:23.165271 31773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:23.160563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.3:59581 (local address 127.13.100.1:39575)
0929 18:41:23.160685 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0929 18:41:23.160690 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:41:23.161018 (+ 328us) server_negotiation.cc:407] Connection header received
0929 18:41:23.161077 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:41:23.161079 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:41:23.161140 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:41:23.161218 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:41:23.161636 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:23.162452 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:23.164048 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:23.164678 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:23.164792 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:23.164834 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0929 18:41:23.165096 (+ 262us) server_negotiation.cc:299] Negotiation successful
0929 18:41:23.165178 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 18:41:23.165494 31582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5313922f223418ea6cbf4670e69a13a" candidate_uuid: "0101fca067db410cb59233fd2436b9c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82929379546a4715b81ee33bd65b5d73" is_pre_election: true
I20260929 18:41:23.165623 31785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:23.160455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.1:39575 (local address 127.13.100.3:59581)
0929 18:41:23.160891 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:23.160906 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 18:41:23.161004 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:41:23.161241 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:41:23.161244 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:41:23.161282 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:41:23.161503 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:41:23.161512 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:41:23.162597 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:41:23.162604 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:41:23.163911 (+ 1307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:41:23.163922 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:41:23.164803 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:41:23.164810 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:41:23.164914 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:23.164948 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 18:41:23.165442 (+ 494us) client_negotiation.cc:240] Negotiation successful
0929 18:41:23.165514 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":363,"mutex_wait_us":77,"spinlock_wait_cycles":28800,"thread_start_us":132,"threads_started":1}
I20260929 18:41:23.165843 31582 raft_consensus.cc:2454] T c5313922f223418ea6cbf4670e69a13a P 82929379546a4715b81ee33bd65b5d73 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0101fca067db410cb59233fd2436b9c2 in term 0.
I20260929 18:41:23.166462 31646 leader_election.cc:302] T c5313922f223418ea6cbf4670e69a13a P 0101fca067db410cb59233fd2436b9c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0101fca067db410cb59233fd2436b9c2, 82929379546a4715b81ee33bd65b5d73; no voters:
I20260929 18:41:23.166764 31782 raft_consensus.cc:2781] T c5313922f223418ea6cbf4670e69a13a P 0101fca067db410cb59233fd2436b9c2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:41:23.166735 31451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5313922f223418ea6cbf4670e69a13a" candidate_uuid: "0101fca067db410cb59233fd2436b9c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4830904637f8434bbf706ad9092d4e1b" is_pre_election: true
I20260929 18:41:23.166855 31782 raft_consensus.cc:482] T c5313922f223418ea6cbf4670e69a13a P 0101fca067db410cb59233fd2436b9c2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:41:23.166908 31782 raft_consensus.cc:3037] T c5313922f223418ea6cbf4670e69a13a P 0101fca067db410cb59233fd2436b9c2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:41:23.167016 31451 raft_consensus.cc:2454] T c5313922f223418ea6cbf4670e69a13a P 4830904637f8434bbf706ad9092d4e1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0101fca067db410cb59233fd2436b9c2 in term 0.
I20260929 18:41:23.168105 31782 raft_consensus.cc:504] T c5313922f223418ea6cbf4670e69a13a P 0101fca067db410cb59233fd2436b9c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0101fca067db410cb59233fd2436b9c2" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 46597 } } peers { permanent_uuid: "4830904637f8434bbf706ad9092d4e1b" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 39575 } } peers { permanent_uuid: "82929379546a4715b81ee33bd65b5d73" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 33939 } }
I20260929 18:41:23.168449 31782 leader_election.cc:288] T c5313922f223418ea6cbf4670e69a13a P 0101fca067db410cb59233fd2436b9c2 [CANDIDATE]: Term 1 election: Requested vote from peers 4830904637f8434bbf706ad9092d4e1b (127.13.100.1:39575), 82929379546a4715b81ee33bd65b5d73 (127.13.100.2:33939)
I20260929 18:41:23.168723 31451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5313922f223418ea6cbf4670e69a13a" candidate_uuid: "0101fca067db410cb59233fd2436b9c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4830904637f8434bbf706ad9092d4e1b"
I20260929 18:41:23.168815 31582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5313922f223418ea6cbf4670e69a13a" candidate_uuid: "0101fca067db410cb59233fd2436b9c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82929379546a4715b81ee33bd65b5d73"
I20260929 18:41:23.168862 31451 raft_consensus.cc:3037] T c5313922f223418ea6cbf4670e69a13a P 4830904637f8434bbf706ad9092d4e1b [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:41:23.169009 31582 raft_consensus.cc:3037] T c5313922f223418ea6cbf4670e69a13a P 82929379546a4715b81ee33bd65b5d73 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:41:23.170187 31451 raft_consensus.cc:2454] T c5313922f223418ea6cbf4670e69a13a P 4830904637f8434bbf706ad9092d4e1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0101fca067db410cb59233fd2436b9c2 in term 1.
I20260929 18:41:23.170252 31582 raft_consensus.cc:2454] T c5313922f223418ea6cbf4670e69a13a P 82929379546a4715b81ee33bd65b5d73 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0101fca067db410cb59233fd2436b9c2 in term 1.
I20260929 18:41:23.170526 31647 leader_election.cc:302] T c5313922f223418ea6cbf4670e69a13a P 0101fca067db410cb59233fd2436b9c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0101fca067db410cb59233fd2436b9c2, 4830904637f8434bbf706ad9092d4e1b; no voters:
I20260929 18:41:23.170725 31782 raft_consensus.cc:2781] T c5313922f223418ea6cbf4670e69a13a P 0101fca067db410cb59233fd2436b9c2 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:41:23.170923 31782 raft_consensus.cc:686] T c5313922f223418ea6cbf4670e69a13a P 0101fca067db410cb59233fd2436b9c2 [term 1 LEADER]: Becoming Leader. State: Replica: 0101fca067db410cb59233fd2436b9c2, State: Running, Role: LEADER
I20260929 18:41:23.171248 31782 consensus_queue.cc:237] T c5313922f223418ea6cbf4670e69a13a P 0101fca067db410cb59233fd2436b9c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0101fca067db410cb59233fd2436b9c2" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 46597 } } peers { permanent_uuid: "4830904637f8434bbf706ad9092d4e1b" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 39575 } } peers { permanent_uuid: "82929379546a4715b81ee33bd65b5d73" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 33939 } }
I20260929 18:41:23.174338 31308 catalog_manager.cc:5495] T c5313922f223418ea6cbf4670e69a13a P 0101fca067db410cb59233fd2436b9c2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0101fca067db410cb59233fd2436b9c2 (127.13.100.3). New cstate: current_term: 1 leader_uuid: "0101fca067db410cb59233fd2436b9c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0101fca067db410cb59233fd2436b9c2" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 46597 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4830904637f8434bbf706ad9092d4e1b" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 39575 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82929379546a4715b81ee33bd65b5d73" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 33939 } health_report { overall_health: UNKNOWN } } }
I20260929 18:41:23.248706 31772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:23.245189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36242 (local address 127.13.100.3:46597)
0929 18:41:23.245340 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0929 18:41:23.245346 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:41:23.245362 (+ 16us) server_negotiation.cc:407] Connection header received
0929 18:41:23.245463 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:41:23.245466 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:41:23.245519 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:41:23.245637 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:41:23.245897 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:23.246404 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:23.247259 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:23.247487 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:23.247600 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:23.247908 (+ 308us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:41:23.248315 (+ 407us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:41:23.248448 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0929 18:41:23.248510 (+ 62us) server_negotiation.cc:299] Negotiation successful
0929 18:41:23.248564 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 18:41:23.254426 31451 raft_consensus.cc:1261] T c5313922f223418ea6cbf4670e69a13a P 4830904637f8434bbf706ad9092d4e1b [term 1 FOLLOWER]: Refusing update from remote peer 0101fca067db410cb59233fd2436b9c2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:41:23.254426 31582 raft_consensus.cc:1261] T c5313922f223418ea6cbf4670e69a13a P 82929379546a4715b81ee33bd65b5d73 [term 1 FOLLOWER]: Refusing update from remote peer 0101fca067db410cb59233fd2436b9c2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:41:23.255089 31782 consensus_queue.cc:1035] T c5313922f223418ea6cbf4670e69a13a P 0101fca067db410cb59233fd2436b9c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82929379546a4715b81ee33bd65b5d73" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 33939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:41:23.255252 31786 consensus_queue.cc:1035] T c5313922f223418ea6cbf4670e69a13a P 0101fca067db410cb59233fd2436b9c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4830904637f8434bbf706ad9092d4e1b" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 39575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:41:23.263123 31791 mvcc.cc:205] Tried to move back new op lower bound from 7334737032202600448 to 7334737031872303104. Current Snapshot: MvccSnapshot[applied={T|T < 7334737032202600448}]
I20260929 18:41:23.265303 31790 mvcc.cc:205] Tried to move back new op lower bound from 7334737032202600448 to 7334737031872303104. Current Snapshot: MvccSnapshot[applied={T|T < 7334737032202600448}]
I20260929 18:41:23.270023 31308 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50756:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:41:23.270157 31308 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50756:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:41:23.272418 31308 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 30a3748deec14a3a9527730ef2e9d290: Sending DeleteTablet for 3 replicas of tablet c5313922f223418ea6cbf4670e69a13a
I20260929 18:41:23.273113 31431 tablet_service.cc:1514] Processing DeleteTablet for tablet c5313922f223418ea6cbf4670e69a13a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:41:23 UTC) from {username='slave'} at 127.0.0.1:35208
I20260929 18:41:23.273109 31693 tablet_service.cc:1514] Processing DeleteTablet for tablet c5313922f223418ea6cbf4670e69a13a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:41:23 UTC) from {username='slave'} at 127.0.0.1:36236
I20260929 18:41:23.273267 31562 tablet_service.cc:1514] Processing DeleteTablet for tablet c5313922f223418ea6cbf4670e69a13a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:41:23 UTC) from {username='slave'} at 127.0.0.1:51374
I20260929 18:41:23.273582 31800 tablet_replica.cc:330] T c5313922f223418ea6cbf4670e69a13a P 4830904637f8434bbf706ad9092d4e1b: stopping tablet replica
I20260929 18:41:23.273871 31800 raft_consensus.cc:2229] T c5313922f223418ea6cbf4670e69a13a P 4830904637f8434bbf706ad9092d4e1b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:41:23.273924 31799 tablet_replica.cc:330] T c5313922f223418ea6cbf4670e69a13a P 0101fca067db410cb59233fd2436b9c2: stopping tablet replica
I20260929 18:41:23.274080 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 31369
I20260929 18:41:23.275264 31799 raft_consensus.cc:2229] T c5313922f223418ea6cbf4670e69a13a P 0101fca067db410cb59233fd2436b9c2 [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:41:23.275573 31799 raft_consensus.cc:2258] T c5313922f223418ea6cbf4670e69a13a P 0101fca067db410cb59233fd2436b9c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:41:23.276752 31799 ts_tablet_manager.cc:1905] T c5313922f223418ea6cbf4670e69a13a P 0101fca067db410cb59233fd2436b9c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:41:23.278579 31801 tablet_replica.cc:330] T c5313922f223418ea6cbf4670e69a13a P 82929379546a4715b81ee33bd65b5d73: stopping tablet replica
I20260929 18:41:23.278914 31801 raft_consensus.cc:2229] T c5313922f223418ea6cbf4670e69a13a P 82929379546a4715b81ee33bd65b5d73 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:41:23.279199 31799 ts_tablet_manager.cc:1918] T c5313922f223418ea6cbf4670e69a13a P 0101fca067db410cb59233fd2436b9c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:41:23.279302 31799 log.cc:1196] T c5313922f223418ea6cbf4670e69a13a P 0101fca067db410cb59233fd2436b9c2: Deleting WAL directory at /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707036041078-13712-0/minicluster-data/ts-2/wal/wals/c5313922f223418ea6cbf4670e69a13a
I20260929 18:41:23.279423 31801 raft_consensus.cc:2258] T c5313922f223418ea6cbf4670e69a13a P 82929379546a4715b81ee33bd65b5d73 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:41:23.279632 31799 ts_tablet_manager.cc:1939] T c5313922f223418ea6cbf4670e69a13a P 0101fca067db410cb59233fd2436b9c2: Deleting consensus metadata
I20260929 18:41:23.280821 31296 catalog_manager.cc:4841] TS 0101fca067db410cb59233fd2436b9c2 (127.13.100.3:46597): tablet c5313922f223418ea6cbf4670e69a13a (table test-table [id=d536ae43709443d788edfd00fd1d8b89]) successfully deleted
I20260929 18:41:23.281343 31801 ts_tablet_manager.cc:1905] T c5313922f223418ea6cbf4670e69a13a P 82929379546a4715b81ee33bd65b5d73: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260929 18:41:23.281572 31294 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.100.1:39575 (error 108)
I20260929 18:41:23.282619 31771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:23.282311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.1:39575 (local address 127.0.0.1:35220)
0929 18:41:23.282450 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:23.282530 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.1:39575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260929 18:41:23.282642 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 31500
W20260929 18:41:23.282882 31294 catalog_manager.cc:4584] TS 4830904637f8434bbf706ad9092d4e1b (127.13.100.1:39575): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c5313922f223418ea6cbf4670e69a13a: Network error: Client connection negotiation failed: client connection to 127.13.100.1:39575: connect: Connection refused (error 111)
I20260929 18:41:23.291551 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 31631
I20260929 18:41:23.291749 31771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:23.291473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.2:33939 (local address 127.0.0.1:51382)
0929 18:41:23.291615 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:23.291669 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.2:33939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260929 18:41:23.298000 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 31278
2026-09-29T18:41:23Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1295 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707036041078-13712-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:41:23 dist-test-slave-j1xs krb5kdc[31805](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:41:23 dist-test-slave-j1xs krb5kdc[31805](info): set up 2 sockets
Sep 29 18:41:23 dist-test-slave-j1xs krb5kdc[31805](info): commencing operation
krb5kdc: starting...
W20260929 18:41:24.338269 13712 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707036041078-13712-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707036041078-13712-0/krb5kdc/test-user.keytab.
Sep 29 18:41:24 dist-test-slave-j1xs krb5kdc[31805](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707284, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:41:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:41:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.100.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.100.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.100.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.62.keytab.
Entry for principal kudu/127.13.100.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.62.keytab.
I20260929 18:41:24.458668 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707036041078-13712-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707036041078-13712-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707036041078-13712-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.100.62:43071
--webserver_interface=127.13.100.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.62.keytab
--principal=kudu/127.13.100.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.100.20:39539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.100.62:43071
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707036041078-13712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707036041078-13712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707036041078-13712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:41:24.571798 31818 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:41:24.572033 31818 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:41:24.572072 31818 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:41:24.576078 31818 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:41:24.576128 31818 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:41:24.576146 31818 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:41:24.576186 31818 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:41:24.576223 31818 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:41:24.581341 31818 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.100.20:39539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707036041078-13712-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707036041078-13712-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.100.62:43071
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.62.keytab
--principal=kudu/127.13.100.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.100.62:43071
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.100.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31818
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707036041078-13712-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:41:24.582391 31818 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:41:24.583364 31818 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:41:24.591782 31824 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:41:24.591806 31826 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:41:24.591789 31823 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:41:24.592211 31818 server_base.cc:1034] running on GCE node
I20260929 18:41:24.592800 31818 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:41:24.593823 31818 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:41:24.595039 31818 hybrid_clock.cc:648] HybridClock initialized: now 1790707284595014 us; error 40 us; skew 500 ppm
Sep 29 18:41:24 dist-test-slave-j1xs krb5kdc[31805](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707284, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:41:24.597867 31818 init.cc:377] Logged in from keytab as kudu/127.13.100.62@KRBTEST.COM (short username kudu)
I20260929 18:41:24.598918 31818 webserver.cc:460] Webserver started at http://127.13.100.62:40487/ using document root <none> and password file <none>
I20260929 18:41:24.599472 31818 fs_manager.cc:359] Metadata directory not provided
I20260929 18:41:24.599565 31818 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:41:24.599792 31818 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:41:24.601779 31818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707036041078-13712-0/minicluster-data/master-0/data/instance:
uuid: "c998e567de9e4ffca09aadbc01b6593a"
format_stamp: "Formatted at 2026-09-29 18:41:24 on dist-test-slave-j1xs"
server_key: "69802ed80d85915f77fc44da7b27d98f"
server_key_iv: "db49975bbfc878a4ba188082db82e742"
server_key_version: "encryptionkey@0"
I20260929 18:41:24.602388 31818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707036041078-13712-0/minicluster-data/master-0/wal/instance:
uuid: "c998e567de9e4ffca09aadbc01b6593a"
format_stamp: "Formatted at 2026-09-29 18:41:24 on dist-test-slave-j1xs"
server_key: "69802ed80d85915f77fc44da7b27d98f"
server_key_iv: "db49975bbfc878a4ba188082db82e742"
server_key_version: "encryptionkey@0"
I20260929 18:41:24.605840 31818 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260929 18:41:24.608131 31833 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:41:24.609340 31818 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:41:24.609486 31818 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707036041078-13712-0/minicluster-data/master-0/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707036041078-13712-0/minicluster-data/master-0/wal
uuid: "c998e567de9e4ffca09aadbc01b6593a"
format_stamp: "Formatted at 2026-09-29 18:41:24 on dist-test-slave-j1xs"
server_key: "69802ed80d85915f77fc44da7b27d98f"
server_key_iv: "db49975bbfc878a4ba188082db82e742"
server_key_version: "encryptionkey@0"
I20260929 18:41:24.609606 31818 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707036041078-13712-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707036041078-13712-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707036041078-13712-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:41:24.623237 31818 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:41:24.626173 31818 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:41:24.626376 31818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:41:24.633177 31818 rpc_server.cc:307] RPC server started. Bound to: 127.13.100.62:43071
I20260929 18:41:24.633229 31885 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.100.62:43071 every 8 connection(s)
I20260929 18:41:24.634186 31818 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
I20260929 18:41:24.634706 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 31818
I20260929 18:41:24.634866 13712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707036041078-13712-0/minicluster-data/master-0/wal/instance
I20260929 18:41:24.635182 13712 external_mini_cluster.cc:1428] Setting key 43aa04f227afbb755dd66ef0510df3a5
I20260929 18:41:24.637174 31886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:41:24.643455 31886 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c998e567de9e4ffca09aadbc01b6593a: Bootstrap starting.
Sep 29 18:41:24 dist-test-slave-j1xs krb5kdc[31805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707284, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.100.62@KRBTEST.COM
I20260929 18:41:24.645951 31886 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c998e567de9e4ffca09aadbc01b6593a: Neither blocks nor log segments found. Creating new log.
I20260929 18:41:24.646817 31886 log.cc:824] T 00000000000000000000000000000000 P c998e567de9e4ffca09aadbc01b6593a: Log is configured to *not* fsync() on all Append() calls
I20260929 18:41:24.648962 31886 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c998e567de9e4ffca09aadbc01b6593a: No bootstrap required, opened a new log
I20260929 18:41:24.649652 31889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:24.636510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42470 (local address 127.13.100.62:43071)
0929 18:41:24.636903 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0929 18:41:24.636923 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 18:41:24.636963 (+ 40us) server_negotiation.cc:407] Connection header received
0929 18:41:24.637629 (+ 666us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:41:24.637644 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:41:24.638026 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:41:24.638288 (+ 262us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:41:24.639161 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:24.640156 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:24.641291 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:24.641752 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:24.641873 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:24.644573 (+ 2700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:41:24.644581 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:41:24.644598 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:41:24.644628 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:41:24.646609 (+ 1981us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:41:24.647123 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:41:24.647130 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:41:24.647137 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:41:24.647271 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:41:24.647598 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:41:24.647600 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:41:24.647602 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:41:24.648064 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:41:24.648245 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0929 18:41:24.648512 (+ 267us) server_negotiation.cc:299] Negotiation successful
0929 18:41:24.648785 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":134,"threads_started":1}
I20260929 18:41:24.652777 31886 raft_consensus.cc:348] T 00000000000000000000000000000000 P c998e567de9e4ffca09aadbc01b6593a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c998e567de9e4ffca09aadbc01b6593a" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 43071 } }
I20260929 18:41:24.652971 31886 raft_consensus.cc:374] T 00000000000000000000000000000000 P c998e567de9e4ffca09aadbc01b6593a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:41:24.653004 31886 raft_consensus.cc:729] T 00000000000000000000000000000000 P c998e567de9e4ffca09aadbc01b6593a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c998e567de9e4ffca09aadbc01b6593a, State: Initialized, Role: FOLLOWER
I20260929 18:41:24.653396 31886 consensus_queue.cc:260] T 00000000000000000000000000000000 P c998e567de9e4ffca09aadbc01b6593a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c998e567de9e4ffca09aadbc01b6593a" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 43071 } }
I20260929 18:41:24.653517 31886 raft_consensus.cc:388] T 00000000000000000000000000000000 P c998e567de9e4ffca09aadbc01b6593a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:41:24.653555 31886 raft_consensus.cc:482] T 00000000000000000000000000000000 P c998e567de9e4ffca09aadbc01b6593a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:41:24.653618 31886 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c998e567de9e4ffca09aadbc01b6593a [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:41:24.654433 31886 raft_consensus.cc:504] T 00000000000000000000000000000000 P c998e567de9e4ffca09aadbc01b6593a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c998e567de9e4ffca09aadbc01b6593a" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 43071 } }
I20260929 18:41:24.654706 31886 leader_election.cc:302] T 00000000000000000000000000000000 P c998e567de9e4ffca09aadbc01b6593a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c998e567de9e4ffca09aadbc01b6593a; no voters:
I20260929 18:41:24.654929 31886 leader_election.cc:288] T 00000000000000000000000000000000 P c998e567de9e4ffca09aadbc01b6593a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:41:24.655074 31891 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c998e567de9e4ffca09aadbc01b6593a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:41:24.655391 31891 raft_consensus.cc:686] T 00000000000000000000000000000000 P c998e567de9e4ffca09aadbc01b6593a [term 1 LEADER]: Becoming Leader. State: Replica: c998e567de9e4ffca09aadbc01b6593a, State: Running, Role: LEADER
I20260929 18:41:24.655665 31891 consensus_queue.cc:237] T 00000000000000000000000000000000 P c998e567de9e4ffca09aadbc01b6593a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c998e567de9e4ffca09aadbc01b6593a" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 43071 } }
I20260929 18:41:24.656104 31886 sys_catalog.cc:564] T 00000000000000000000000000000000 P c998e567de9e4ffca09aadbc01b6593a [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:41:24.657313 31893 sys_catalog.cc:455] T 00000000000000000000000000000000 P c998e567de9e4ffca09aadbc01b6593a [sys.catalog]: SysCatalogTable state changed. Reason: New leader c998e567de9e4ffca09aadbc01b6593a. Latest consensus state: current_term: 1 leader_uuid: "c998e567de9e4ffca09aadbc01b6593a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c998e567de9e4ffca09aadbc01b6593a" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 43071 } } }
I20260929 18:41:24.657431 31893 sys_catalog.cc:458] T 00000000000000000000000000000000 P c998e567de9e4ffca09aadbc01b6593a [sys.catalog]: This master's current role is: LEADER
I20260929 18:41:24.657698 31892 sys_catalog.cc:455] T 00000000000000000000000000000000 P c998e567de9e4ffca09aadbc01b6593a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c998e567de9e4ffca09aadbc01b6593a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c998e567de9e4ffca09aadbc01b6593a" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 43071 } } }
I20260929 18:41:24.657809 31892 sys_catalog.cc:458] T 00000000000000000000000000000000 P c998e567de9e4ffca09aadbc01b6593a [sys.catalog]: This master's current role is: LEADER
I20260929 18:41:24.658468 31900 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:41:24.661165 31900 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:41:24.666484 31900 catalog_manager.cc:1347] Generated new cluster ID: 75f0bbd3b44d458fa092a5215910e68f
I20260929 18:41:24.666543 31900 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:41:24.677649 31900 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:41:24.678864 31900 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:41:24.685837 31900 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c998e567de9e4ffca09aadbc01b6593a: Generated new TSK 0
I20260929 18:41:24.686492 31900 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.100.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.100.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.100.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.1.keytab.
Entry for principal kudu/127.13.100.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.1.keytab.
I20260929 18:41:24.719237 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707036041078-13712-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707036041078-13712-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707036041078-13712-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707036041078-13712-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.100.1:0
--local_ip_for_outbound_sockets=127.13.100.1
--webserver_interface=127.13.100.1
--webserver_port=0
--tserver_master_addrs=127.13.100.62:43071
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.1.keytab
--principal=kudu/127.13.100.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.100.20:39539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707036041078-13712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707036041078-13712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707036041078-13712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:41:24.829838 31912 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:41:24.830127 31912 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:41:24.830170 31912 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:41:24.834241 31912 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:41:24.834306 31912 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:41:24.834363 31912 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.100.1
I20260929 18:41:24.839932 31912 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.100.20:39539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707036041078-13712-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707036041078-13712-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.1.keytab
--principal=kudu/127.13.100.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.100.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707036041078-13712-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.100.1
--webserver_port=0
--tserver_master_addrs=127.13.100.62:43071
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31912
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.100.1
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707036041078-13712-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:41:24.841014 31912 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:41:24.842044 31912 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:41:24.851346 31917 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:41:24.851354 31918 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:41:24.851409 31920 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:41:24.851570 31912 server_base.cc:1034] running on GCE node
I20260929 18:41:24.852042 31912 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:41:24.852655 31912 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:41:24.853971 31912 hybrid_clock.cc:648] HybridClock initialized: now 1790707284853955 us; error 34 us; skew 500 ppm
Sep 29 18:41:24 dist-test-slave-j1xs krb5kdc[31805](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707284, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:41:24.859474 31912 init.cc:377] Logged in from keytab as kudu/127.13.100.1@KRBTEST.COM (short username kudu)
I20260929 18:41:24.860334 31912 webserver.cc:460] Webserver started at http://127.13.100.1:37909/ using document root <none> and password file <none>
I20260929 18:41:24.860838 31912 fs_manager.cc:359] Metadata directory not provided
I20260929 18:41:24.860895 31912 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:41:24.861055 31912 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:41:24.863019 31912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707036041078-13712-0/minicluster-data/ts-0/data/instance:
uuid: "fe06e585c32143cf94214af08c269eac"
format_stamp: "Formatted at 2026-09-29 18:41:24 on dist-test-slave-j1xs"
server_key: "be100a7bc1f3dbd3f25aacf1ced3cf0b"
server_key_iv: "49a111d98b54220204de2fd0cf352f68"
server_key_version: "encryptionkey@0"
I20260929 18:41:24.863511 31912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707036041078-13712-0/minicluster-data/ts-0/wal/instance:
uuid: "fe06e585c32143cf94214af08c269eac"
format_stamp: "Formatted at 2026-09-29 18:41:24 on dist-test-slave-j1xs"
server_key: "be100a7bc1f3dbd3f25aacf1ced3cf0b"
server_key_iv: "49a111d98b54220204de2fd0cf352f68"
server_key_version: "encryptionkey@0"
I20260929 18:41:24.866999 31912 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 18:41:24.869308 31927 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:41:24.870635 31912 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:41:24.870774 31912 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707036041078-13712-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707036041078-13712-0/minicluster-data/ts-0/wal
uuid: "fe06e585c32143cf94214af08c269eac"
format_stamp: "Formatted at 2026-09-29 18:41:24 on dist-test-slave-j1xs"
server_key: "be100a7bc1f3dbd3f25aacf1ced3cf0b"
server_key_iv: "49a111d98b54220204de2fd0cf352f68"
server_key_version: "encryptionkey@0"
I20260929 18:41:24.870878 31912 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707036041078-13712-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707036041078-13712-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707036041078-13712-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:41:24.897024 31912 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:41:24.900437 31912 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:41:24.900655 31912 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:41:24.901269 31912 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:41:24.902136 31912 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:41:24.902184 31912 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:41:24.902252 31912 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:41:24.902302 31912 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:41:24.911305 31912 rpc_server.cc:307] RPC server started. Bound to: 127.13.100.1:40883
I20260929 18:41:24.911326 32040 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.100.1:40883 every 8 connection(s)
I20260929 18:41:24.912156 31912 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707036041078-13712-0/minicluster-data/ts-0/data/info.pb
I20260929 18:41:24.916074 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 31912
I20260929 18:41:24.916220 13712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707036041078-13712-0/minicluster-data/ts-0/wal/instance
I20260929 18:41:24.916479 13712 external_mini_cluster.cc:1428] Setting key 943a2051ebd9f1f9d87086dbe4f9e521
Sep 29 18:41:24 dist-test-slave-j1xs krb5kdc[31805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707284, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.1@KRBTEST.COM for kudu/127.13.100.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.13.100.2@KRBTEST.COM; defaulting to no policy
I20260929 18:41:24.926776 31889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:24.913974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.1:49607 (local address 127.13.100.62:43071)
0929 18:41:24.914118 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0929 18:41:24.914122 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:41:24.914945 (+ 823us) server_negotiation.cc:407] Connection header received
0929 18:41:24.916233 (+ 1288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:41:24.916236 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:41:24.916291 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:41:24.916370 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:41:24.917796 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:24.918613 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:24.919924 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:24.920185 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:24.920309 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:24.923114 (+ 2805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:41:24.923117 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:41:24.923119 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:41:24.923139 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:41:24.924583 (+ 1444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:41:24.925193 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:41:24.925195 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:41:24.925197 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:41:24.925265 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:41:24.925635 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:41:24.925639 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:41:24.925641 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:41:24.925830 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:41:24.925970 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0929 18:41:24.926461 (+ 491us) server_negotiation.cc:299] Negotiation successful
0929 18:41:24.926655 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 18:41:24.927554 32043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:24.914343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:43071 (local address 127.13.100.1:49607)
0929 18:41:24.914785 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:24.914836 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 18:41:24.916033 (+ 1197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:41:24.916524 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:41:24.916531 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:41:24.916988 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:41:24.917603 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:41:24.917623 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:41:24.918741 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:41:24.918744 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:41:24.919801 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:41:24.919811 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:41:24.920344 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:41:24.920349 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:41:24.920508 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:24.921135 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:41:24.921150 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:41:24.922938 (+ 1788us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:41:24.924700 (+ 1762us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:41:24.924707 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:41:24.924720 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:41:24.925096 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:41:24.925350 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:41:24.925353 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:41:24.925355 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:41:24.925526 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:41:24.925996 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:41:24.926000 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:41:24.926219 (+ 219us) client_negotiation.cc:769] Sending connection context
0929 18:41:24.926371 (+ 152us) client_negotiation.cc:240] Negotiation successful
0929 18:41:24.926691 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":107,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.100.2@KRBTEST.COM" created.
I20260929 18:41:24.928925 32041 heartbeater.cc:347] Connected to a master server at 127.13.100.62:43071
I20260929 18:41:24.929224 32041 heartbeater.cc:464] Registering TS with master...
I20260929 18:41:24.929844 32041 heartbeater.cc:511] Master 127.13.100.62:43071 requested a full tablet report, sending...
I20260929 18:41:24.931463 31849 ts_manager.cc:194] Registered new tserver with Master: fe06e585c32143cf94214af08c269eac (127.13.100.1:40883)
I20260929 18:41:24.932657 31849 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.100.1@KRBTEST.COM'} at 127.13.100.1:49607
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.100.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.2.keytab.
Entry for principal kudu/127.13.100.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.2.keytab.
I20260929 18:41:24.943056 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707036041078-13712-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707036041078-13712-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707036041078-13712-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707036041078-13712-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.100.2:0
--local_ip_for_outbound_sockets=127.13.100.2
--webserver_interface=127.13.100.2
--webserver_port=0
--tserver_master_addrs=127.13.100.62:43071
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.2.keytab
--principal=kudu/127.13.100.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.100.20:39539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707036041078-13712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707036041078-13712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707036041078-13712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:41:25.050706 32046 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:41:25.050935 32046 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:41:25.051020 32046 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:41:25.055018 32046 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:41:25.055073 32046 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:41:25.055179 32046 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.100.2
I20260929 18:41:25.060729 32046 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.100.20:39539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707036041078-13712-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707036041078-13712-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.2.keytab
--principal=kudu/127.13.100.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.100.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707036041078-13712-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.100.2
--webserver_port=0
--tserver_master_addrs=127.13.100.62:43071
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32046
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.100.2
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707036041078-13712-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:41:25.061744 32046 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:41:25.062774 32046 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:41:25.072184 32051 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:41:25.072173 32052 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:41:25.072183 32054 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:41:25.072456 32046 server_base.cc:1034] running on GCE node
I20260929 18:41:25.072889 32046 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:41:25.073531 32046 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:41:25.074728 32046 hybrid_clock.cc:648] HybridClock initialized: now 1790707285074711 us; error 28 us; skew 500 ppm
Sep 29 18:41:25 dist-test-slave-j1xs krb5kdc[31805](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707285, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:41:25.077359 32046 init.cc:377] Logged in from keytab as kudu/127.13.100.2@KRBTEST.COM (short username kudu)
I20260929 18:41:25.078205 32046 webserver.cc:460] Webserver started at http://127.13.100.2:39265/ using document root <none> and password file <none>
I20260929 18:41:25.078707 32046 fs_manager.cc:359] Metadata directory not provided
I20260929 18:41:25.078764 32046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:41:25.078922 32046 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:41:25.080869 32046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707036041078-13712-0/minicluster-data/ts-1/data/instance:
uuid: "c416834ffa6a45539aa145e1b5ade4df"
format_stamp: "Formatted at 2026-09-29 18:41:25 on dist-test-slave-j1xs"
server_key: "cf6615d95e52d77cc62057d1f0fec92d"
server_key_iv: "30f48042d55d5e0501054826a9d028f7"
server_key_version: "encryptionkey@0"
I20260929 18:41:25.081372 32046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707036041078-13712-0/minicluster-data/ts-1/wal/instance:
uuid: "c416834ffa6a45539aa145e1b5ade4df"
format_stamp: "Formatted at 2026-09-29 18:41:25 on dist-test-slave-j1xs"
server_key: "cf6615d95e52d77cc62057d1f0fec92d"
server_key_iv: "30f48042d55d5e0501054826a9d028f7"
server_key_version: "encryptionkey@0"
I20260929 18:41:25.084897 32046 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260929 18:41:25.087266 32061 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:41:25.088486 32046 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 18:41:25.088632 32046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707036041078-13712-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707036041078-13712-0/minicluster-data/ts-1/wal
uuid: "c416834ffa6a45539aa145e1b5ade4df"
format_stamp: "Formatted at 2026-09-29 18:41:25 on dist-test-slave-j1xs"
server_key: "cf6615d95e52d77cc62057d1f0fec92d"
server_key_iv: "30f48042d55d5e0501054826a9d028f7"
server_key_version: "encryptionkey@0"
I20260929 18:41:25.088716 32046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707036041078-13712-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707036041078-13712-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707036041078-13712-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:41:25.106868 32046 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:41:25.110140 32046 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:41:25.110318 32046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:41:25.110899 32046 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:41:25.111662 32046 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:41:25.111708 32046 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:41:25.111747 32046 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:41:25.111763 32046 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:41:25.120909 32046 rpc_server.cc:307] RPC server started. Bound to: 127.13.100.2:43439
I20260929 18:41:25.120927 32174 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.100.2:43439 every 8 connection(s)
I20260929 18:41:25.121814 32046 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707036041078-13712-0/minicluster-data/ts-1/data/info.pb
I20260929 18:41:25.129025 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 32046
I20260929 18:41:25.129190 13712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707036041078-13712-0/minicluster-data/ts-1/wal/instance
I20260929 18:41:25.129551 13712 external_mini_cluster.cc:1428] Setting key e54c3ff37478fd56ec0a7dfbdad4e307
Sep 29 18:41:25 dist-test-slave-j1xs krb5kdc[31805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707285, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.2@KRBTEST.COM for kudu/127.13.100.62@KRBTEST.COM
I20260929 18:41:25.140615 31889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:25.123745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.2:37251 (local address 127.13.100.62:43071)
0929 18:41:25.123988 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0929 18:41:25.123992 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:41:25.124766 (+ 774us) server_negotiation.cc:407] Connection header received
0929 18:41:25.126149 (+ 1383us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:41:25.126153 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:41:25.126218 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:41:25.126310 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:41:25.127637 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:25.128575 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:25.130732 (+ 2157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:25.131027 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:25.131145 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:25.135541 (+ 4396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:41:25.135548 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:41:25.135553 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:41:25.135586 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:41:25.137592 (+ 2006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:41:25.138396 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:41:25.138437 (+ 41us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:41:25.138442 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:41:25.138575 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:41:25.139180 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:41:25.139184 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:41:25.139186 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:41:25.139449 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:41:25.139542 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 18:41:25.140337 (+ 795us) server_negotiation.cc:299] Negotiation successful
0929 18:41:25.140448 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150}
WARNING: no policy specified for kudu/127.13.100.3@KRBTEST.COM; defaulting to no policy
I20260929 18:41:25.142073 32177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:25.124166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:43071 (local address 127.13.100.2:37251)
0929 18:41:25.124619 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:25.124660 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 18:41:25.125865 (+ 1205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:41:25.126482 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:41:25.126489 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:41:25.126846 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:41:25.127429 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:41:25.127453 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:41:25.128757 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:41:25.128761 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:41:25.130491 (+ 1730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:41:25.130506 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:41:25.131197 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:41:25.131205 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:41:25.131439 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:25.132295 (+ 856us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:41:25.132316 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:41:25.135299 (+ 2983us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:41:25.137741 (+ 2442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:41:25.137747 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:41:25.137766 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:41:25.138202 (+ 436us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:41:25.138791 (+ 589us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:41:25.138795 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:41:25.138799 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:41:25.139042 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:41:25.139620 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:41:25.139630 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:41:25.140017 (+ 387us) client_negotiation.cc:769] Sending connection context
0929 18:41:25.140268 (+ 251us) client_negotiation.cc:240] Negotiation successful
0929 18:41:25.140966 (+ 698us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":120,"threads_started":1}
I20260929 18:41:25.143265 32175 heartbeater.cc:347] Connected to a master server at 127.13.100.62:43071
I20260929 18:41:25.143566 32175 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.100.3@KRBTEST.COM" created.
I20260929 18:41:25.144163 32175 heartbeater.cc:511] Master 127.13.100.62:43071 requested a full tablet report, sending...
I20260929 18:41:25.145283 31849 ts_manager.cc:194] Registered new tserver with Master: c416834ffa6a45539aa145e1b5ade4df (127.13.100.2:43439)
I20260929 18:41:25.145992 31849 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.100.2@KRBTEST.COM'} at 127.13.100.2:37251
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.100.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.3.keytab.
Entry for principal kudu/127.13.100.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.3.keytab.
I20260929 18:41:25.161353 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707036041078-13712-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707036041078-13712-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707036041078-13712-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707036041078-13712-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.100.3:0
--local_ip_for_outbound_sockets=127.13.100.3
--webserver_interface=127.13.100.3
--webserver_port=0
--tserver_master_addrs=127.13.100.62:43071
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.3.keytab
--principal=kudu/127.13.100.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.100.20:39539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707036041078-13712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707036041078-13712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707036041078-13712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:41:25.272964 32180 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:41:25.273269 32180 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:41:25.273355 32180 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:41:25.277536 32180 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:41:25.277626 32180 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:41:25.277714 32180 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.100.3
I20260929 18:41:25.283049 32180 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.100.20:39539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707036041078-13712-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707036041078-13712-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.3.keytab
--principal=kudu/127.13.100.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.100.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707036041078-13712-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.100.3
--webserver_port=0
--tserver_master_addrs=127.13.100.62:43071
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32180
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.100.3
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707036041078-13712-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:41:25.284113 32180 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:41:25.285126 32180 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:41:25.296120 32186 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:41:25.296175 32180 server_base.cc:1034] running on GCE node
W20260929 18:41:25.296314 32188 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:41:25.296156 32185 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:41:25.296842 32180 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:41:25.297508 32180 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:41:25.298707 32180 hybrid_clock.cc:648] HybridClock initialized: now 1790707285298686 us; error 35 us; skew 500 ppm
Sep 29 18:41:25 dist-test-slave-j1xs krb5kdc[31805](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707285, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:41:25.303236 32180 init.cc:377] Logged in from keytab as kudu/127.13.100.3@KRBTEST.COM (short username kudu)
I20260929 18:41:25.304301 32180 webserver.cc:460] Webserver started at http://127.13.100.3:43297/ using document root <none> and password file <none>
I20260929 18:41:25.304927 32180 fs_manager.cc:359] Metadata directory not provided
I20260929 18:41:25.305027 32180 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:41:25.305251 32180 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:41:25.307458 32180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707036041078-13712-0/minicluster-data/ts-2/data/instance:
uuid: "9bcfe2c619124d64aebf75cf841fc2f1"
format_stamp: "Formatted at 2026-09-29 18:41:25 on dist-test-slave-j1xs"
server_key: "b2020148b41976cc6cb554f2a7a31129"
server_key_iv: "f90d1bba0d7551e3ded8e45cd3026718"
server_key_version: "encryptionkey@0"
I20260929 18:41:25.308043 32180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707036041078-13712-0/minicluster-data/ts-2/wal/instance:
uuid: "9bcfe2c619124d64aebf75cf841fc2f1"
format_stamp: "Formatted at 2026-09-29 18:41:25 on dist-test-slave-j1xs"
server_key: "b2020148b41976cc6cb554f2a7a31129"
server_key_iv: "f90d1bba0d7551e3ded8e45cd3026718"
server_key_version: "encryptionkey@0"
I20260929 18:41:25.312405 32180 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 18:41:25.315529 32195 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:41:25.316898 32180 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 18:41:25.317050 32180 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707036041078-13712-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707036041078-13712-0/minicluster-data/ts-2/wal
uuid: "9bcfe2c619124d64aebf75cf841fc2f1"
format_stamp: "Formatted at 2026-09-29 18:41:25 on dist-test-slave-j1xs"
server_key: "b2020148b41976cc6cb554f2a7a31129"
server_key_iv: "f90d1bba0d7551e3ded8e45cd3026718"
server_key_version: "encryptionkey@0"
I20260929 18:41:25.317180 32180 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707036041078-13712-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707036041078-13712-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707036041078-13712-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:41:25.342378 32180 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:41:25.345342 32180 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:41:25.345542 32180 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:41:25.346148 32180 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:41:25.346971 32180 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:41:25.347041 32180 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:41:25.347106 32180 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:41:25.347148 32180 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:41:25.356112 32180 rpc_server.cc:307] RPC server started. Bound to: 127.13.100.3:37463
I20260929 18:41:25.356164 32308 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.100.3:37463 every 8 connection(s)
I20260929 18:41:25.357012 32180 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707036041078-13712-0/minicluster-data/ts-2/data/info.pb
I20260929 18:41:25.358461 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 32180
I20260929 18:41:25.358618 13712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707036041078-13712-0/minicluster-data/ts-2/wal/instance
I20260929 18:41:25.358951 13712 external_mini_cluster.cc:1428] Setting key 98282b629e335ce6469f7ed88d893b03
Sep 29 18:41:25 dist-test-slave-j1xs krb5kdc[31805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707285, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.3@KRBTEST.COM for kudu/127.13.100.62@KRBTEST.COM
I20260929 18:41:25.370158 31889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:25.359080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.3:44925 (local address 127.13.100.62:43071)
0929 18:41:25.359223 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0929 18:41:25.359229 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:41:25.360117 (+ 888us) server_negotiation.cc:407] Connection header received
0929 18:41:25.361423 (+ 1306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:41:25.361425 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:41:25.361468 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:41:25.361533 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:41:25.362676 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:25.363195 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:25.364263 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:25.364474 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:25.364552 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:25.366874 (+ 2322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:41:25.366877 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:41:25.366879 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:41:25.366900 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:41:25.368321 (+ 1421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:41:25.368784 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:41:25.368788 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:41:25.368790 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:41:25.368884 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:41:25.369195 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:41:25.369197 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:41:25.369199 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:41:25.369361 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:41:25.369447 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 18:41:25.369937 (+ 490us) server_negotiation.cc:299] Negotiation successful
0929 18:41:25.370042 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 18:41:25.370910 32311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:25.359510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:43071 (local address 127.13.100.3:44925)
0929 18:41:25.359960 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:25.360010 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 18:41:25.361248 (+ 1238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:41:25.361687 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:41:25.361694 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:41:25.362109 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:41:25.362554 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:41:25.362567 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:41:25.363305 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:41:25.363308 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:41:25.364157 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:41:25.364166 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:41:25.364596 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:41:25.364598 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:41:25.364755 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:25.365295 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:41:25.365310 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:41:25.366731 (+ 1421us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:41:25.368422 (+ 1691us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:41:25.368426 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:41:25.368436 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:41:25.368674 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:41:25.368984 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:41:25.368986 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:41:25.368987 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:41:25.369113 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:41:25.369445 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:41:25.369450 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:41:25.369687 (+ 237us) client_negotiation.cc:769] Sending connection context
0929 18:41:25.369845 (+ 158us) client_negotiation.cc:240] Negotiation successful
0929 18:41:25.370135 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":122,"threads_started":1}
I20260929 18:41:25.372123 32309 heartbeater.cc:347] Connected to a master server at 127.13.100.62:43071
I20260929 18:41:25.372381 32309 heartbeater.cc:464] Registering TS with master...
I20260929 18:41:25.372876 32309 heartbeater.cc:511] Master 127.13.100.62:43071 requested a full tablet report, sending...
I20260929 18:41:25.373842 31850 ts_manager.cc:194] Registered new tserver with Master: 9bcfe2c619124d64aebf75cf841fc2f1 (127.13.100.3:37463)
I20260929 18:41:25.374502 31850 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.100.3@KRBTEST.COM'} at 127.13.100.3:44925
I20260929 18:41:25.384372 13712 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 18:41:25.394053 31889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:25.386345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42480 (local address 127.13.100.62:43071)
0929 18:41:25.386469 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0929 18:41:25.386473 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:41:25.386647 (+ 174us) server_negotiation.cc:407] Connection header received
0929 18:41:25.386752 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:41:25.386754 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:41:25.386790 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:41:25.386857 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:41:25.387547 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:25.388040 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:25.388850 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:25.389087 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:25.389161 (+ 74us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:25.390788 (+ 1627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:41:25.390790 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:41:25.390792 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:41:25.390814 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:41:25.392123 (+ 1309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:41:25.392673 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:41:25.392677 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:41:25.392681 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:41:25.392791 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:41:25.393137 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:41:25.393142 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:41:25.393146 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:41:25.393364 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:41:25.393524 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0929 18:41:25.393782 (+ 258us) server_negotiation.cc:299] Negotiation successful
0929 18:41:25.393873 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 18:41:25.396740 31850 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42480:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:41:25.399644 31850 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:41:25.413491 32320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:25.407301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.2:43439 (local address 127.0.0.1:33474)
0929 18:41:25.408010 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:25.408055 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 18:41:25.408201 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:41:25.408943 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:41:25.408949 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:41:25.408969 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:41:25.409140 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:41:25.409151 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:41:25.410740 (+ 1589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:41:25.410743 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:41:25.411909 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:41:25.411917 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:41:25.412744 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:41:25.412747 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:41:25.412937 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:25.413080 (+ 143us) client_negotiation.cc:769] Sending connection context
0929 18:41:25.413187 (+ 107us) client_negotiation.cc:240] Negotiation successful
0929 18:41:25.413319 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"mutex_wait_us":103,"spinlock_wait_cycles":6272,"thread_start_us":54,"threads_started":1}
I20260929 18:41:25.413594 32325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:25.408052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58986 (local address 127.13.100.3:37463)
0929 18:41:25.408526 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0929 18:41:25.408541 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 18:41:25.408844 (+ 303us) server_negotiation.cc:407] Connection header received
0929 18:41:25.408919 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:41:25.408924 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:41:25.409080 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:41:25.409229 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:41:25.409430 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:25.410642 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:25.412142 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:25.412985 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:25.413098 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:25.413237 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0929 18:41:25.413366 (+ 129us) server_negotiation.cc:299] Negotiation successful
0929 18:41:25.413450 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":151,"threads_started":1}
I20260929 18:41:25.413815 32324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:25.408013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39828 (local address 127.13.100.1:40883)
0929 18:41:25.408474 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0929 18:41:25.408490 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 18:41:25.408514 (+ 24us) server_negotiation.cc:407] Connection header received
0929 18:41:25.408585 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:41:25.408590 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:41:25.408784 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:41:25.408929 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:41:25.409382 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:25.410213 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:25.411800 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:25.412632 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:25.412737 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:25.412881 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0929 18:41:25.413557 (+ 676us) server_negotiation.cc:299] Negotiation successful
0929 18:41:25.413671 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":101,"threads_started":1}
I20260929 18:41:25.414068 32321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:25.407396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.1:40883 (local address 127.0.0.1:39828)
0929 18:41:25.408059 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:25.408073 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:41:25.408203 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:41:25.409108 (+ 905us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:41:25.409110 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:41:25.409125 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:41:25.409252 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:41:25.409258 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:41:25.410334 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:41:25.410340 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:41:25.411671 (+ 1331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:41:25.411681 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:41:25.412957 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:41:25.412960 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:41:25.413104 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:25.413146 (+ 42us) client_negotiation.cc:769] Sending connection context
0929 18:41:25.413355 (+ 209us) client_negotiation.cc:240] Negotiation successful
0929 18:41:25.413432 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":592,"mutex_wait_us":32,"spinlock_wait_cycles":3712,"thread_start_us":85,"threads_started":1}
I20260929 18:41:25.414233 32322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:25.407409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33474 (local address 127.13.100.2:43439)
0929 18:41:25.408473 (+ 1064us) server_negotiation.cc:206] Beginning negotiation
0929 18:41:25.408485 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 18:41:25.408506 (+ 21us) server_negotiation.cc:407] Connection header received
0929 18:41:25.408585 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:41:25.408590 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:41:25.408783 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:41:25.409276 (+ 493us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:41:25.409446 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:25.410599 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:25.412022 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:25.412613 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:25.412735 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:25.412878 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0929 18:41:25.413830 (+ 952us) server_negotiation.cc:299] Negotiation successful
0929 18:41:25.414100 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":967,"thread_start_us":102,"threads_started":1}
I20260929 18:41:25.413489 32323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:25.407677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.3:37463 (local address 127.0.0.1:58986)
0929 18:41:25.408662 (+ 985us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:25.408674 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:41:25.408754 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:41:25.409212 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:41:25.409218 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:41:25.409234 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:41:25.409320 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:41:25.409328 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:41:25.410774 (+ 1446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:41:25.410777 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:41:25.412016 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:41:25.412026 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:41:25.413101 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:41:25.413105 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:41:25.413235 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:25.413255 (+ 20us) client_negotiation.cc:769] Sending connection context
0929 18:41:25.413322 (+ 67us) client_negotiation.cc:240] Negotiation successful
0929 18:41:25.413372 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":931,"mutex_wait_us":195,"thread_start_us":86,"threads_started":1}
I20260929 18:41:25.416652 31975 tablet_service.cc:1467] Processing CreateTablet for tablet 9f2c5e8884d74931b98f441c8f3e0980 (DEFAULT_TABLE table=test-table [id=d3c18aba23eb4b969f66754c78206dd0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:41:25.416814 32243 tablet_service.cc:1467] Processing CreateTablet for tablet 9f2c5e8884d74931b98f441c8f3e0980 (DEFAULT_TABLE table=test-table [id=d3c18aba23eb4b969f66754c78206dd0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:41:25.416814 32109 tablet_service.cc:1467] Processing CreateTablet for tablet 9f2c5e8884d74931b98f441c8f3e0980 (DEFAULT_TABLE table=test-table [id=d3c18aba23eb4b969f66754c78206dd0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:41:25.417542 31975 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f2c5e8884d74931b98f441c8f3e0980. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:41:25.417760 32109 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f2c5e8884d74931b98f441c8f3e0980. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:41:25.417764 32243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f2c5e8884d74931b98f441c8f3e0980. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:41:25.424033 32326 tablet_bootstrap.cc:492] T 9f2c5e8884d74931b98f441c8f3e0980 P fe06e585c32143cf94214af08c269eac: Bootstrap starting.
I20260929 18:41:25.424996 32327 tablet_bootstrap.cc:492] T 9f2c5e8884d74931b98f441c8f3e0980 P c416834ffa6a45539aa145e1b5ade4df: Bootstrap starting.
I20260929 18:41:25.426216 32328 tablet_bootstrap.cc:492] T 9f2c5e8884d74931b98f441c8f3e0980 P 9bcfe2c619124d64aebf75cf841fc2f1: Bootstrap starting.
I20260929 18:41:25.426491 32326 tablet_bootstrap.cc:654] T 9f2c5e8884d74931b98f441c8f3e0980 P fe06e585c32143cf94214af08c269eac: Neither blocks nor log segments found. Creating new log.
I20260929 18:41:25.427286 32327 tablet_bootstrap.cc:654] T 9f2c5e8884d74931b98f441c8f3e0980 P c416834ffa6a45539aa145e1b5ade4df: Neither blocks nor log segments found. Creating new log.
I20260929 18:41:25.427542 32326 log.cc:824] T 9f2c5e8884d74931b98f441c8f3e0980 P fe06e585c32143cf94214af08c269eac: Log is configured to *not* fsync() on all Append() calls
I20260929 18:41:25.428067 32327 log.cc:824] T 9f2c5e8884d74931b98f441c8f3e0980 P c416834ffa6a45539aa145e1b5ade4df: Log is configured to *not* fsync() on all Append() calls
I20260929 18:41:25.428256 32328 tablet_bootstrap.cc:654] T 9f2c5e8884d74931b98f441c8f3e0980 P 9bcfe2c619124d64aebf75cf841fc2f1: Neither blocks nor log segments found. Creating new log.
I20260929 18:41:25.429386 32328 log.cc:824] T 9f2c5e8884d74931b98f441c8f3e0980 P 9bcfe2c619124d64aebf75cf841fc2f1: Log is configured to *not* fsync() on all Append() calls
I20260929 18:41:25.429836 32327 tablet_bootstrap.cc:492] T 9f2c5e8884d74931b98f441c8f3e0980 P c416834ffa6a45539aa145e1b5ade4df: No bootstrap required, opened a new log
I20260929 18:41:25.429831 32326 tablet_bootstrap.cc:492] T 9f2c5e8884d74931b98f441c8f3e0980 P fe06e585c32143cf94214af08c269eac: No bootstrap required, opened a new log
I20260929 18:41:25.430040 32326 ts_tablet_manager.cc:1397] T 9f2c5e8884d74931b98f441c8f3e0980 P fe06e585c32143cf94214af08c269eac: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260929 18:41:25.430040 32327 ts_tablet_manager.cc:1397] T 9f2c5e8884d74931b98f441c8f3e0980 P c416834ffa6a45539aa145e1b5ade4df: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260929 18:41:25.431006 32328 tablet_bootstrap.cc:492] T 9f2c5e8884d74931b98f441c8f3e0980 P 9bcfe2c619124d64aebf75cf841fc2f1: No bootstrap required, opened a new log
I20260929 18:41:25.431217 32328 ts_tablet_manager.cc:1397] T 9f2c5e8884d74931b98f441c8f3e0980 P 9bcfe2c619124d64aebf75cf841fc2f1: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 18:41:25.433763 32327 raft_consensus.cc:348] T 9f2c5e8884d74931b98f441c8f3e0980 P c416834ffa6a45539aa145e1b5ade4df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c416834ffa6a45539aa145e1b5ade4df" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 43439 } } peers { permanent_uuid: "fe06e585c32143cf94214af08c269eac" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 40883 } } peers { permanent_uuid: "9bcfe2c619124d64aebf75cf841fc2f1" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 37463 } }
I20260929 18:41:25.433763 32326 raft_consensus.cc:348] T 9f2c5e8884d74931b98f441c8f3e0980 P fe06e585c32143cf94214af08c269eac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c416834ffa6a45539aa145e1b5ade4df" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 43439 } } peers { permanent_uuid: "fe06e585c32143cf94214af08c269eac" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 40883 } } peers { permanent_uuid: "9bcfe2c619124d64aebf75cf841fc2f1" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 37463 } }
I20260929 18:41:25.434046 32326 raft_consensus.cc:374] T 9f2c5e8884d74931b98f441c8f3e0980 P fe06e585c32143cf94214af08c269eac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:41:25.434046 32327 raft_consensus.cc:374] T 9f2c5e8884d74931b98f441c8f3e0980 P c416834ffa6a45539aa145e1b5ade4df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:41:25.434105 32327 raft_consensus.cc:729] T 9f2c5e8884d74931b98f441c8f3e0980 P c416834ffa6a45539aa145e1b5ade4df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c416834ffa6a45539aa145e1b5ade4df, State: Initialized, Role: FOLLOWER
I20260929 18:41:25.434105 32326 raft_consensus.cc:729] T 9f2c5e8884d74931b98f441c8f3e0980 P fe06e585c32143cf94214af08c269eac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe06e585c32143cf94214af08c269eac, State: Initialized, Role: FOLLOWER
I20260929 18:41:25.433864 32328 raft_consensus.cc:348] T 9f2c5e8884d74931b98f441c8f3e0980 P 9bcfe2c619124d64aebf75cf841fc2f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c416834ffa6a45539aa145e1b5ade4df" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 43439 } } peers { permanent_uuid: "fe06e585c32143cf94214af08c269eac" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 40883 } } peers { permanent_uuid: "9bcfe2c619124d64aebf75cf841fc2f1" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 37463 } }
I20260929 18:41:25.434181 32328 raft_consensus.cc:374] T 9f2c5e8884d74931b98f441c8f3e0980 P 9bcfe2c619124d64aebf75cf841fc2f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:41:25.434238 32328 raft_consensus.cc:729] T 9f2c5e8884d74931b98f441c8f3e0980 P 9bcfe2c619124d64aebf75cf841fc2f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bcfe2c619124d64aebf75cf841fc2f1, State: Initialized, Role: FOLLOWER
I20260929 18:41:25.434544 32327 consensus_queue.cc:260] T 9f2c5e8884d74931b98f441c8f3e0980 P c416834ffa6a45539aa145e1b5ade4df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c416834ffa6a45539aa145e1b5ade4df" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 43439 } } peers { permanent_uuid: "fe06e585c32143cf94214af08c269eac" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 40883 } } peers { permanent_uuid: "9bcfe2c619124d64aebf75cf841fc2f1" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 37463 } }
I20260929 18:41:25.434544 32326 consensus_queue.cc:260] T 9f2c5e8884d74931b98f441c8f3e0980 P fe06e585c32143cf94214af08c269eac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c416834ffa6a45539aa145e1b5ade4df" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 43439 } } peers { permanent_uuid: "fe06e585c32143cf94214af08c269eac" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 40883 } } peers { permanent_uuid: "9bcfe2c619124d64aebf75cf841fc2f1" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 37463 } }
I20260929 18:41:25.434624 32328 consensus_queue.cc:260] T 9f2c5e8884d74931b98f441c8f3e0980 P 9bcfe2c619124d64aebf75cf841fc2f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c416834ffa6a45539aa145e1b5ade4df" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 43439 } } peers { permanent_uuid: "fe06e585c32143cf94214af08c269eac" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 40883 } } peers { permanent_uuid: "9bcfe2c619124d64aebf75cf841fc2f1" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 37463 } }
I20260929 18:41:25.435237 32309 heartbeater.cc:503] Master 127.13.100.62:43071 was elected leader, sending a full tablet report...
I20260929 18:41:25.435456 32327 ts_tablet_manager.cc:1428] T 9f2c5e8884d74931b98f441c8f3e0980 P c416834ffa6a45539aa145e1b5ade4df: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260929 18:41:25.435456 32326 ts_tablet_manager.cc:1428] T 9f2c5e8884d74931b98f441c8f3e0980 P fe06e585c32143cf94214af08c269eac: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260929 18:41:25.435506 32328 ts_tablet_manager.cc:1428] T 9f2c5e8884d74931b98f441c8f3e0980 P 9bcfe2c619124d64aebf75cf841fc2f1: Time spent starting tablet: real 0.004s user 0.000s sys 0.005s
I20260929 18:41:25.435760 32041 heartbeater.cc:503] Master 127.13.100.62:43071 was elected leader, sending a full tablet report...
I20260929 18:41:25.436579 32175 heartbeater.cc:503] Master 127.13.100.62:43071 was elected leader, sending a full tablet report...
I20260929 18:41:25.496023 32333 raft_consensus.cc:482] T 9f2c5e8884d74931b98f441c8f3e0980 P c416834ffa6a45539aa145e1b5ade4df [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:41:25.496258 32333 raft_consensus.cc:504] T 9f2c5e8884d74931b98f441c8f3e0980 P c416834ffa6a45539aa145e1b5ade4df [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c416834ffa6a45539aa145e1b5ade4df" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 43439 } } peers { permanent_uuid: "fe06e585c32143cf94214af08c269eac" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 40883 } } peers { permanent_uuid: "9bcfe2c619124d64aebf75cf841fc2f1" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 37463 } }
I20260929 18:41:25.497390 32333 leader_election.cc:288] T 9f2c5e8884d74931b98f441c8f3e0980 P c416834ffa6a45539aa145e1b5ade4df [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe06e585c32143cf94214af08c269eac (127.13.100.1:40883), 9bcfe2c619124d64aebf75cf841fc2f1 (127.13.100.3:37463)
I20260929 18:41:25.501514 32177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:25.497647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.1:40883 (local address 127.13.100.2:51619)
0929 18:41:25.497776 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:25.497793 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 18:41:25.497915 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:41:25.498278 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:41:25.498282 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:41:25.498307 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:41:25.498440 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:41:25.498449 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:41:25.499362 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:41:25.499365 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:41:25.500267 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:41:25.500276 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:41:25.501046 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:41:25.501050 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:41:25.501171 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:25.501197 (+ 26us) client_negotiation.cc:769] Sending connection context
0929 18:41:25.501280 (+ 83us) client_negotiation.cc:240] Negotiation successful
0929 18:41:25.501366 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260929 18:41:25.501583 32324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:25.497860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.2:51619 (local address 127.13.100.1:40883)
0929 18:41:25.498016 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0929 18:41:25.498020 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:41:25.498035 (+ 15us) server_negotiation.cc:407] Connection header received
0929 18:41:25.498103 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:41:25.498105 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:41:25.498161 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:41:25.498244 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:41:25.498601 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:25.499230 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:25.500396 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:25.500933 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:25.501031 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:25.501072 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0929 18:41:25.501356 (+ 284us) server_negotiation.cc:299] Negotiation successful
0929 18:41:25.501434 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260929 18:41:25.501941 32325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:25.497796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.2:52633 (local address 127.13.100.3:37463)
0929 18:41:25.497946 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0929 18:41:25.497951 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:41:25.498255 (+ 304us) server_negotiation.cc:407] Connection header received
0929 18:41:25.498353 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:41:25.498355 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:41:25.498404 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:41:25.498487 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:41:25.498964 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:25.499583 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:25.500767 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:25.501281 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:25.501387 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:25.501429 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0929 18:41:25.501725 (+ 296us) server_negotiation.cc:299] Negotiation successful
0929 18:41:25.501801 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 18:41:25.502270 32335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:25.497647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.3:37463 (local address 127.13.100.2:52633)
0929 18:41:25.498135 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:25.498148 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:41:25.498254 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:41:25.498593 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:41:25.498596 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:41:25.498616 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:41:25.498743 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:41:25.498749 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:41:25.499708 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:41:25.499711 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:41:25.500641 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:41:25.500650 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:41:25.501403 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:41:25.501406 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:41:25.501559 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:25.501582 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 18:41:25.502102 (+ 520us) client_negotiation.cc:240] Negotiation successful
0929 18:41:25.502174 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":430,"mutex_wait_us":139,"thread_start_us":145,"threads_started":1}
I20260929 18:41:25.502254 31995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f2c5e8884d74931b98f441c8f3e0980" candidate_uuid: "c416834ffa6a45539aa145e1b5ade4df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe06e585c32143cf94214af08c269eac" is_pre_election: true
I20260929 18:41:25.502580 31995 raft_consensus.cc:2454] T 9f2c5e8884d74931b98f441c8f3e0980 P fe06e585c32143cf94214af08c269eac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c416834ffa6a45539aa145e1b5ade4df in term 0.
I20260929 18:41:25.502892 32263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f2c5e8884d74931b98f441c8f3e0980" candidate_uuid: "c416834ffa6a45539aa145e1b5ade4df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bcfe2c619124d64aebf75cf841fc2f1" is_pre_election: true
I20260929 18:41:25.503135 32064 leader_election.cc:302] T 9f2c5e8884d74931b98f441c8f3e0980 P c416834ffa6a45539aa145e1b5ade4df [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c416834ffa6a45539aa145e1b5ade4df, fe06e585c32143cf94214af08c269eac; no voters:
I20260929 18:41:25.503178 32263 raft_consensus.cc:2454] T 9f2c5e8884d74931b98f441c8f3e0980 P 9bcfe2c619124d64aebf75cf841fc2f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c416834ffa6a45539aa145e1b5ade4df in term 0.
I20260929 18:41:25.503449 32333 raft_consensus.cc:2781] T 9f2c5e8884d74931b98f441c8f3e0980 P c416834ffa6a45539aa145e1b5ade4df [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:41:25.503546 32333 raft_consensus.cc:482] T 9f2c5e8884d74931b98f441c8f3e0980 P c416834ffa6a45539aa145e1b5ade4df [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:41:25.503597 32333 raft_consensus.cc:3037] T 9f2c5e8884d74931b98f441c8f3e0980 P c416834ffa6a45539aa145e1b5ade4df [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:41:25.504657 32333 raft_consensus.cc:504] T 9f2c5e8884d74931b98f441c8f3e0980 P c416834ffa6a45539aa145e1b5ade4df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c416834ffa6a45539aa145e1b5ade4df" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 43439 } } peers { permanent_uuid: "fe06e585c32143cf94214af08c269eac" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 40883 } } peers { permanent_uuid: "9bcfe2c619124d64aebf75cf841fc2f1" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 37463 } }
I20260929 18:41:25.505018 32333 leader_election.cc:288] T 9f2c5e8884d74931b98f441c8f3e0980 P c416834ffa6a45539aa145e1b5ade4df [CANDIDATE]: Term 1 election: Requested vote from peers fe06e585c32143cf94214af08c269eac (127.13.100.1:40883), 9bcfe2c619124d64aebf75cf841fc2f1 (127.13.100.3:37463)
I20260929 18:41:25.505357 31995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f2c5e8884d74931b98f441c8f3e0980" candidate_uuid: "c416834ffa6a45539aa145e1b5ade4df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe06e585c32143cf94214af08c269eac"
I20260929 18:41:25.505461 32263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f2c5e8884d74931b98f441c8f3e0980" candidate_uuid: "c416834ffa6a45539aa145e1b5ade4df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bcfe2c619124d64aebf75cf841fc2f1"
I20260929 18:41:25.505501 31995 raft_consensus.cc:3037] T 9f2c5e8884d74931b98f441c8f3e0980 P fe06e585c32143cf94214af08c269eac [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:41:25.505637 32263 raft_consensus.cc:3037] T 9f2c5e8884d74931b98f441c8f3e0980 P 9bcfe2c619124d64aebf75cf841fc2f1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:41:25.506680 31995 raft_consensus.cc:2454] T 9f2c5e8884d74931b98f441c8f3e0980 P fe06e585c32143cf94214af08c269eac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c416834ffa6a45539aa145e1b5ade4df in term 1.
I20260929 18:41:25.506719 32263 raft_consensus.cc:2454] T 9f2c5e8884d74931b98f441c8f3e0980 P 9bcfe2c619124d64aebf75cf841fc2f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c416834ffa6a45539aa145e1b5ade4df in term 1.
I20260929 18:41:25.507045 32063 leader_election.cc:302] T 9f2c5e8884d74931b98f441c8f3e0980 P c416834ffa6a45539aa145e1b5ade4df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bcfe2c619124d64aebf75cf841fc2f1, c416834ffa6a45539aa145e1b5ade4df; no voters:
I20260929 18:41:25.507284 32333 raft_consensus.cc:2781] T 9f2c5e8884d74931b98f441c8f3e0980 P c416834ffa6a45539aa145e1b5ade4df [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:41:25.507516 32333 raft_consensus.cc:686] T 9f2c5e8884d74931b98f441c8f3e0980 P c416834ffa6a45539aa145e1b5ade4df [term 1 LEADER]: Becoming Leader. State: Replica: c416834ffa6a45539aa145e1b5ade4df, State: Running, Role: LEADER
I20260929 18:41:25.507848 32333 consensus_queue.cc:237] T 9f2c5e8884d74931b98f441c8f3e0980 P c416834ffa6a45539aa145e1b5ade4df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c416834ffa6a45539aa145e1b5ade4df" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 43439 } } peers { permanent_uuid: "fe06e585c32143cf94214af08c269eac" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 40883 } } peers { permanent_uuid: "9bcfe2c619124d64aebf75cf841fc2f1" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 37463 } }
I20260929 18:41:25.511307 31849 catalog_manager.cc:5495] T 9f2c5e8884d74931b98f441c8f3e0980 P c416834ffa6a45539aa145e1b5ade4df reported cstate change: term changed from 0 to 1, leader changed from <none> to c416834ffa6a45539aa145e1b5ade4df (127.13.100.2). New cstate: current_term: 1 leader_uuid: "c416834ffa6a45539aa145e1b5ade4df" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c416834ffa6a45539aa145e1b5ade4df" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 43439 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe06e585c32143cf94214af08c269eac" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 40883 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9bcfe2c619124d64aebf75cf841fc2f1" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 37463 } health_report { overall_health: UNKNOWN } } }
I20260929 18:41:25.535933 32322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:25.532486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33478 (local address 127.13.100.2:43439)
0929 18:41:25.532610 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0929 18:41:25.532614 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:41:25.532645 (+ 31us) server_negotiation.cc:407] Connection header received
0929 18:41:25.532711 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:41:25.532713 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:41:25.532761 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:41:25.532847 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:41:25.533037 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:25.533512 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:25.534322 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:25.534526 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:25.534602 (+ 76us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:25.535004 (+ 402us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:41:25.535399 (+ 395us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:41:25.535515 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0929 18:41:25.535672 (+ 157us) server_negotiation.cc:299] Negotiation successful
0929 18:41:25.535747 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 18:41:25.541317 32263 raft_consensus.cc:1261] T 9f2c5e8884d74931b98f441c8f3e0980 P 9bcfe2c619124d64aebf75cf841fc2f1 [term 1 FOLLOWER]: Refusing update from remote peer c416834ffa6a45539aa145e1b5ade4df: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:41:25.541334 31995 raft_consensus.cc:1261] T 9f2c5e8884d74931b98f441c8f3e0980 P fe06e585c32143cf94214af08c269eac [term 1 FOLLOWER]: Refusing update from remote peer c416834ffa6a45539aa145e1b5ade4df: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:41:25.541998 32336 consensus_queue.cc:1035] T 9f2c5e8884d74931b98f441c8f3e0980 P c416834ffa6a45539aa145e1b5ade4df [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe06e585c32143cf94214af08c269eac" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 40883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:41:25.542193 32333 consensus_queue.cc:1035] T 9f2c5e8884d74931b98f441c8f3e0980 P c416834ffa6a45539aa145e1b5ade4df [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bcfe2c619124d64aebf75cf841fc2f1" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 37463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:41:25.548710 32339 mvcc.cc:205] Tried to move back new op lower bound from 7334737041570635776 to 7334737041442881536. Current Snapshot: MvccSnapshot[applied={T|T < 7334737041570635776}]
I20260929 18:41:25.549538 32341 mvcc.cc:205] Tried to move back new op lower bound from 7334737041570635776 to 7334737041442881536. Current Snapshot: MvccSnapshot[applied={T|T < 7334737041570635776}]
I20260929 18:41:25.551991 32340 mvcc.cc:205] Tried to move back new op lower bound from 7334737041570635776 to 7334737041442881536. Current Snapshot: MvccSnapshot[applied={T|T < 7334737041570635776}]
I20260929 18:41:25.557615 31849 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42480:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:41:25.557801 31849 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42480:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:41:25.560322 31849 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c998e567de9e4ffca09aadbc01b6593a: Sending DeleteTablet for 3 replicas of tablet 9f2c5e8884d74931b98f441c8f3e0980
I20260929 18:41:25.561074 32109 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f2c5e8884d74931b98f441c8f3e0980 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:41:25 UTC) from {username='kudu', principal='kudu/127.13.100.62@KRBTEST.COM'} at 127.0.0.1:33474
I20260929 18:41:25.561081 31975 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f2c5e8884d74931b98f441c8f3e0980 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:41:25 UTC) from {username='kudu', principal='kudu/127.13.100.62@KRBTEST.COM'} at 127.0.0.1:39828
I20260929 18:41:25.561187 32243 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f2c5e8884d74931b98f441c8f3e0980 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:41:25 UTC) from {username='kudu', principal='kudu/127.13.100.62@KRBTEST.COM'} at 127.0.0.1:58986
I20260929 18:41:25.561550 32349 tablet_replica.cc:330] T 9f2c5e8884d74931b98f441c8f3e0980 P c416834ffa6a45539aa145e1b5ade4df: stopping tablet replica
I20260929 18:41:25.561841 32350 tablet_replica.cc:330] T 9f2c5e8884d74931b98f441c8f3e0980 P fe06e585c32143cf94214af08c269eac: stopping tablet replica
I20260929 18:41:25.562224 32350 raft_consensus.cc:2229] T 9f2c5e8884d74931b98f441c8f3e0980 P fe06e585c32143cf94214af08c269eac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:41:25.562275 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 31912
I20260929 18:41:25.563014 32351 tablet_replica.cc:330] T 9f2c5e8884d74931b98f441c8f3e0980 P 9bcfe2c619124d64aebf75cf841fc2f1: stopping tablet replica
I20260929 18:41:25.563300 32351 raft_consensus.cc:2229] T 9f2c5e8884d74931b98f441c8f3e0980 P 9bcfe2c619124d64aebf75cf841fc2f1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:41:25.563585 32351 raft_consensus.cc:2258] T 9f2c5e8884d74931b98f441c8f3e0980 P 9bcfe2c619124d64aebf75cf841fc2f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:41:25.564832 32351 ts_tablet_manager.cc:1905] T 9f2c5e8884d74931b98f441c8f3e0980 P 9bcfe2c619124d64aebf75cf841fc2f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:41:25.567020 32351 ts_tablet_manager.cc:1918] T 9f2c5e8884d74931b98f441c8f3e0980 P 9bcfe2c619124d64aebf75cf841fc2f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:41:25.567126 32351 log.cc:1196] T 9f2c5e8884d74931b98f441c8f3e0980 P 9bcfe2c619124d64aebf75cf841fc2f1: Deleting WAL directory at /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707036041078-13712-0/minicluster-data/ts-2/wal/wals/9f2c5e8884d74931b98f441c8f3e0980
I20260929 18:41:25.567461 32351 ts_tablet_manager.cc:1939] T 9f2c5e8884d74931b98f441c8f3e0980 P 9bcfe2c619124d64aebf75cf841fc2f1: Deleting consensus metadata
I20260929 18:41:25.568516 31835 catalog_manager.cc:4841] TS 9bcfe2c619124d64aebf75cf841fc2f1 (127.13.100.3:37463): tablet 9f2c5e8884d74931b98f441c8f3e0980 (table test-table [id=d3c18aba23eb4b969f66754c78206dd0]) successfully deleted
W20260929 18:41:25.569474 31836 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.100.1:40883 (error 108)
I20260929 18:41:25.570519 32323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:25.570244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.1:40883 (local address 127.0.0.1:39840)
0929 18:41:25.570369 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:25.570438 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.1:40883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260929 18:41:25.570694 31836 catalog_manager.cc:4584] TS fe06e585c32143cf94214af08c269eac (127.13.100.1:40883): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9f2c5e8884d74931b98f441c8f3e0980: Network error: Client connection negotiation failed: client connection to 127.13.100.1:40883: connect: Connection refused (error 111)
I20260929 18:41:25.571842 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 32046
I20260929 18:41:25.577029 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 32180
I20260929 18:41:25.578181 32323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:25.577178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.2:43439 (local address 127.0.0.1:33488)
0929 18:41:25.577996 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:25.578065 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.2:43439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":697}
I20260929 18:41:25.582708 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 31818
2026-09-29T18:41:25Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2285 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-29T18:41:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:41:25Z Disabled control of system clock
I20260929 18:41:25.612715 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707036041078-13712-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707036041078-13712-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707036041078-13712-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.100.62:33573
--webserver_interface=127.13.100.62
--webserver_port=0
--builtin_ntp_servers=127.13.100.20:40615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.100.62:33573
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 18:41:25.721220 32356 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:41:25.721493 32356 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:41:25.721529 32356 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:41:25.725637 32356 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:41:25.725708 32356 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:41:25.725725 32356 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:41:25.725742 32356 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:41:25.725757 32356 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:41:25.730836 32356 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.100.20:40615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707036041078-13712-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707036041078-13712-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.100.62:33573
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.100.62:33573
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.100.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32356
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707036041078-13712-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:41:25.731959 32356 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:41:25.733345 32356 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:41:25.741796 32364 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:41:25.741881 32362 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:41:25.741993 32356 server_base.cc:1034] running on GCE node
W20260929 18:41:25.741927 32361 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:41:25.742972 32356 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:41:25.744113 32356 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:41:25.745285 32356 hybrid_clock.cc:648] HybridClock initialized: now 1790707285745255 us; error 45 us; skew 500 ppm
I20260929 18:41:25.747434 32356 webserver.cc:460] Webserver started at http://127.13.100.62:34809/ using document root <none> and password file <none>
I20260929 18:41:25.748198 32356 fs_manager.cc:359] Metadata directory not provided
I20260929 18:41:25.748268 32356 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:41:25.748492 32356 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:41:25.750631 32356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707036041078-13712-0/minicluster-data/master-0/data/instance:
uuid: "5e36295463d640bca4e3d6bfd02f9d84"
format_stamp: "Formatted at 2026-09-29 18:41:25 on dist-test-slave-j1xs"
server_key: "668c9671775e9d5b729dbd57914c1666"
server_key_iv: "e7d72b94e003c578a50f5f784e07da8a"
server_key_version: "encryptionkey@0"
I20260929 18:41:25.751183 32356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707036041078-13712-0/minicluster-data/master-0/wal/instance:
uuid: "5e36295463d640bca4e3d6bfd02f9d84"
format_stamp: "Formatted at 2026-09-29 18:41:25 on dist-test-slave-j1xs"
server_key: "668c9671775e9d5b729dbd57914c1666"
server_key_iv: "e7d72b94e003c578a50f5f784e07da8a"
server_key_version: "encryptionkey@0"
I20260929 18:41:25.755219 32356 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 18:41:25.757611 32370 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:41:25.759078 32356 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:41:25.759224 32356 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707036041078-13712-0/minicluster-data/master-0/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707036041078-13712-0/minicluster-data/master-0/wal
uuid: "5e36295463d640bca4e3d6bfd02f9d84"
format_stamp: "Formatted at 2026-09-29 18:41:25 on dist-test-slave-j1xs"
server_key: "668c9671775e9d5b729dbd57914c1666"
server_key_iv: "e7d72b94e003c578a50f5f784e07da8a"
server_key_version: "encryptionkey@0"
I20260929 18:41:25.759346 32356 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707036041078-13712-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707036041078-13712-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707036041078-13712-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:41:25.772892 32356 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:41:25.773510 32356 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:41:25.773699 32356 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:41:25.780424 32356 rpc_server.cc:307] RPC server started. Bound to: 127.13.100.62:33573
I20260929 18:41:25.780457 32422 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.100.62:33573 every 8 connection(s)
I20260929 18:41:25.781323 32356 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
I20260929 18:41:25.784283 32423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:41:25.788592 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 32356
I20260929 18:41:25.788749 13712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707036041078-13712-0/minicluster-data/master-0/wal/instance
I20260929 18:41:25.789036 13712 external_mini_cluster.cc:1428] Setting key 4ca6bc5b5d74b77158b7977dbb663c4c
I20260929 18:41:25.789604 32423 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e36295463d640bca4e3d6bfd02f9d84: Bootstrap starting.
I20260929 18:41:25.792109 32423 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5e36295463d640bca4e3d6bfd02f9d84: Neither blocks nor log segments found. Creating new log.
I20260929 18:41:25.792958 32423 log.cc:824] T 00000000000000000000000000000000 P 5e36295463d640bca4e3d6bfd02f9d84: Log is configured to *not* fsync() on all Append() calls
I20260929 18:41:25.795056 32423 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e36295463d640bca4e3d6bfd02f9d84: No bootstrap required, opened a new log
I20260929 18:41:25.799455 32426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:25.790439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54656 (local address 127.13.100.62:33573)
0929 18:41:25.791105 (+ 666us) server_negotiation.cc:206] Beginning negotiation
0929 18:41:25.791124 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 18:41:25.791164 (+ 40us) server_negotiation.cc:407] Connection header received
0929 18:41:25.791934 (+ 770us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:41:25.791956 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:41:25.792466 (+ 510us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:41:25.792814 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:41:25.793250 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:25.794639 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:25.795580 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:25.796249 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:25.796418 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:25.797236 (+ 818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:41:25.797242 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:41:25.797257 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:41:25.797737 (+ 480us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:41:25.797805 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:41:25.797829 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:41:25.797957 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0929 18:41:25.798143 (+ 186us) server_negotiation.cc:299] Negotiation successful
0929 18:41:25.798421 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":116,"threads_started":1}
I20260929 18:41:25.799678 32423 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5e36295463d640bca4e3d6bfd02f9d84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e36295463d640bca4e3d6bfd02f9d84" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 33573 } }
I20260929 18:41:25.799908 32423 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5e36295463d640bca4e3d6bfd02f9d84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:41:25.799985 32423 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5e36295463d640bca4e3d6bfd02f9d84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e36295463d640bca4e3d6bfd02f9d84, State: Initialized, Role: FOLLOWER
I20260929 18:41:25.800446 32423 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5e36295463d640bca4e3d6bfd02f9d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e36295463d640bca4e3d6bfd02f9d84" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 33573 } }
I20260929 18:41:25.800624 32423 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5e36295463d640bca4e3d6bfd02f9d84 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:41:25.800691 32423 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5e36295463d640bca4e3d6bfd02f9d84 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:41:25.800782 32423 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5e36295463d640bca4e3d6bfd02f9d84 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:41:25.801914 32423 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5e36295463d640bca4e3d6bfd02f9d84 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e36295463d640bca4e3d6bfd02f9d84" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 33573 } }
I20260929 18:41:25.802205 32423 leader_election.cc:302] T 00000000000000000000000000000000 P 5e36295463d640bca4e3d6bfd02f9d84 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5e36295463d640bca4e3d6bfd02f9d84; no voters:
I20260929 18:41:25.802438 32423 leader_election.cc:288] T 00000000000000000000000000000000 P 5e36295463d640bca4e3d6bfd02f9d84 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:41:25.802565 32428 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5e36295463d640bca4e3d6bfd02f9d84 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:41:25.802861 32428 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5e36295463d640bca4e3d6bfd02f9d84 [term 1 LEADER]: Becoming Leader. State: Replica: 5e36295463d640bca4e3d6bfd02f9d84, State: Running, Role: LEADER
I20260929 18:41:25.803164 32428 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5e36295463d640bca4e3d6bfd02f9d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e36295463d640bca4e3d6bfd02f9d84" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 33573 } }
I20260929 18:41:25.803622 32423 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5e36295463d640bca4e3d6bfd02f9d84 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:41:25.804682 32430 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e36295463d640bca4e3d6bfd02f9d84 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5e36295463d640bca4e3d6bfd02f9d84. Latest consensus state: current_term: 1 leader_uuid: "5e36295463d640bca4e3d6bfd02f9d84" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e36295463d640bca4e3d6bfd02f9d84" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 33573 } } }
I20260929 18:41:25.804802 32430 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e36295463d640bca4e3d6bfd02f9d84 [sys.catalog]: This master's current role is: LEADER
I20260929 18:41:25.805191 32437 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:41:25.806113 32429 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e36295463d640bca4e3d6bfd02f9d84 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5e36295463d640bca4e3d6bfd02f9d84" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e36295463d640bca4e3d6bfd02f9d84" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 33573 } } }
I20260929 18:41:25.806188 32429 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e36295463d640bca4e3d6bfd02f9d84 [sys.catalog]: This master's current role is: LEADER
I20260929 18:41:25.808307 32437 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:41:25.813622 32437 catalog_manager.cc:1347] Generated new cluster ID: 92c30bdc094d4da58823c009b13d9bd1
I20260929 18:41:25.813705 32437 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:41:25.835156 32437 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:41:25.836091 32437 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:41:25.842974 32437 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5e36295463d640bca4e3d6bfd02f9d84: Generated new TSK 0
I20260929 18:41:25.843684 32437 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:41:25.852837 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707036041078-13712-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707036041078-13712-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707036041078-13712-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707036041078-13712-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.100.1:0
--local_ip_for_outbound_sockets=127.13.100.1
--webserver_interface=127.13.100.1
--webserver_port=0
--tserver_master_addrs=127.13.100.62:33573
--builtin_ntp_servers=127.13.100.20:40615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 18:41:25.961469 32447 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:41:25.961701 32447 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:41:25.961737 32447 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:41:25.965932 32447 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:41:25.965977 32447 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:41:25.966076 32447 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.100.1
I20260929 18:41:25.970711 32447 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.100.20:40615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707036041078-13712-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707036041078-13712-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.100.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707036041078-13712-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.100.1
--webserver_port=0
--tserver_master_addrs=127.13.100.62:33573
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32447
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.100.1
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707036041078-13712-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:41:25.971675 32447 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:41:25.972959 32447 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:41:25.981987 32455 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:41:25.981995 32453 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:41:25.981987 32452 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:41:25.982277 32447 server_base.cc:1034] running on GCE node
I20260929 18:41:25.982626 32447 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:41:25.983245 32447 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:41:25.984439 32447 hybrid_clock.cc:648] HybridClock initialized: now 1790707285984427 us; error 29 us; skew 500 ppm
I20260929 18:41:25.986263 32447 webserver.cc:460] Webserver started at http://127.13.100.1:35171/ using document root <none> and password file <none>
I20260929 18:41:25.986799 32447 fs_manager.cc:359] Metadata directory not provided
I20260929 18:41:25.986889 32447 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:41:25.987120 32447 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:41:25.989092 32447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707036041078-13712-0/minicluster-data/ts-0/data/instance:
uuid: "f28cf2df86f0404c838a57df686e4634"
format_stamp: "Formatted at 2026-09-29 18:41:25 on dist-test-slave-j1xs"
server_key: "317b219a3e1728dfc3e157a977fd30ec"
server_key_iv: "8ad17e4851dc9d6ec2f6ca69387ade64"
server_key_version: "encryptionkey@0"
I20260929 18:41:25.989645 32447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707036041078-13712-0/minicluster-data/ts-0/wal/instance:
uuid: "f28cf2df86f0404c838a57df686e4634"
format_stamp: "Formatted at 2026-09-29 18:41:25 on dist-test-slave-j1xs"
server_key: "317b219a3e1728dfc3e157a977fd30ec"
server_key_iv: "8ad17e4851dc9d6ec2f6ca69387ade64"
server_key_version: "encryptionkey@0"
I20260929 18:41:25.993106 32447 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 18:41:25.995497 32461 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:41:25.996621 32447 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260929 18:41:25.996757 32447 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707036041078-13712-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707036041078-13712-0/minicluster-data/ts-0/wal
uuid: "f28cf2df86f0404c838a57df686e4634"
format_stamp: "Formatted at 2026-09-29 18:41:25 on dist-test-slave-j1xs"
server_key: "317b219a3e1728dfc3e157a977fd30ec"
server_key_iv: "8ad17e4851dc9d6ec2f6ca69387ade64"
server_key_version: "encryptionkey@0"
I20260929 18:41:25.996868 32447 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707036041078-13712-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707036041078-13712-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707036041078-13712-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:41:26.018103 32447 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:41:26.018805 32447 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:41:26.019004 32447 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:41:26.019590 32447 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:41:26.020515 32447 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:41:26.020586 32447 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:41:26.020650 32447 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:41:26.020695 32447 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:41:26.029827 32447 rpc_server.cc:307] RPC server started. Bound to: 127.13.100.1:33277
I20260929 18:41:26.029853 32574 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.100.1:33277 every 8 connection(s)
I20260929 18:41:26.030720 32447 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707036041078-13712-0/minicluster-data/ts-0/data/info.pb
I20260929 18:41:26.038946 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 32447
I20260929 18:41:26.039094 13712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707036041078-13712-0/minicluster-data/ts-0/wal/instance
I20260929 18:41:26.039435 13712 external_mini_cluster.cc:1428] Setting key 1b510bb0143d02f5e9cb7d835dd71ac6
I20260929 18:41:26.039834 32426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:26.032495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.1:57845 (local address 127.13.100.62:33573)
0929 18:41:26.032635 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0929 18:41:26.032640 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:41:26.033651 (+ 1011us) server_negotiation.cc:407] Connection header received
0929 18:41:26.035097 (+ 1446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:41:26.035100 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:41:26.035157 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:41:26.035256 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:41:26.036102 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:26.036632 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:26.037758 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:26.038048 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:26.038163 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:26.039277 (+ 1114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:41:26.039280 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:41:26.039282 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:41:26.039294 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:41:26.039312 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:41:26.039321 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:41:26.039395 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 18:41:26.039607 (+ 212us) server_negotiation.cc:299] Negotiation successful
0929 18:41:26.039698 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 18:41:26.040867 32577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:26.032870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:33573 (local address 127.13.100.1:57845)
0929 18:41:26.033480 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:26.033517 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 18:41:26.034870 (+ 1353us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:41:26.035416 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:41:26.035423 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:41:26.035502 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:41:26.035959 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:41:26.035970 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:41:26.036754 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:41:26.036757 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:41:26.037625 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:41:26.037634 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:41:26.038200 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:41:26.038204 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:41:26.038359 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:26.038971 (+ 612us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:41:26.038973 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:41:26.039028 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:41:26.039116 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:41:26.039417 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:41:26.039426 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:41:26.039432 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 18:41:26.039608 (+ 176us) client_negotiation.cc:240] Negotiation successful
0929 18:41:26.039905 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":105,"threads_started":1}
I20260929 18:41:26.041476 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707036041078-13712-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707036041078-13712-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707036041078-13712-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707036041078-13712-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.100.2:0
--local_ip_for_outbound_sockets=127.13.100.2
--webserver_interface=127.13.100.2
--webserver_port=0
--tserver_master_addrs=127.13.100.62:33573
--builtin_ntp_servers=127.13.100.20:40615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 18:41:26.042004 32575 heartbeater.cc:347] Connected to a master server at 127.13.100.62:33573
I20260929 18:41:26.042234 32575 heartbeater.cc:464] Registering TS with master...
I20260929 18:41:26.042824 32575 heartbeater.cc:511] Master 127.13.100.62:33573 requested a full tablet report, sending...
I20260929 18:41:26.044246 32387 ts_manager.cc:194] Registered new tserver with Master: f28cf2df86f0404c838a57df686e4634 (127.13.100.1:33277)
I20260929 18:41:26.045403 32387 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.100.1:57845
W20260929 18:41:26.149966 32578 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:41:26.150243 32578 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:41:26.150313 32578 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:41:26.154412 32578 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:41:26.154508 32578 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:41:26.154600 32578 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.100.2
I20260929 18:41:26.159317 32578 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.100.20:40615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707036041078-13712-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707036041078-13712-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.100.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707036041078-13712-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.100.2
--webserver_port=0
--tserver_master_addrs=127.13.100.62:33573
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32578
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.100.2
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707036041078-13712-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:41:26.160454 32578 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:41:26.161784 32578 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:41:26.171267 32584 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:41:26.171264 32586 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:41:26.171408 32578 server_base.cc:1034] running on GCE node
W20260929 18:41:26.171262 32583 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:41:26.171888 32578 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:41:26.172518 32578 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:41:26.173731 32578 hybrid_clock.cc:648] HybridClock initialized: now 1790707286173704 us; error 46 us; skew 500 ppm
I20260929 18:41:26.175501 32578 webserver.cc:460] Webserver started at http://127.13.100.2:33635/ using document root <none> and password file <none>
I20260929 18:41:26.176085 32578 fs_manager.cc:359] Metadata directory not provided
I20260929 18:41:26.176178 32578 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:41:26.176384 32578 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:41:26.178471 32578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707036041078-13712-0/minicluster-data/ts-1/data/instance:
uuid: "ad2825ca8afe48fb80046d88df4dd9da"
format_stamp: "Formatted at 2026-09-29 18:41:26 on dist-test-slave-j1xs"
server_key: "4af4589a2ec6783a382804f09ba9f841"
server_key_iv: "54d895065170d9f391f2f193b806fcff"
server_key_version: "encryptionkey@0"
I20260929 18:41:26.179014 32578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707036041078-13712-0/minicluster-data/ts-1/wal/instance:
uuid: "ad2825ca8afe48fb80046d88df4dd9da"
format_stamp: "Formatted at 2026-09-29 18:41:26 on dist-test-slave-j1xs"
server_key: "4af4589a2ec6783a382804f09ba9f841"
server_key_iv: "54d895065170d9f391f2f193b806fcff"
server_key_version: "encryptionkey@0"
I20260929 18:41:26.182536 32578 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 18:41:26.184841 32592 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:41:26.186046 32578 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:41:26.186177 32578 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707036041078-13712-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707036041078-13712-0/minicluster-data/ts-1/wal
uuid: "ad2825ca8afe48fb80046d88df4dd9da"
format_stamp: "Formatted at 2026-09-29 18:41:26 on dist-test-slave-j1xs"
server_key: "4af4589a2ec6783a382804f09ba9f841"
server_key_iv: "54d895065170d9f391f2f193b806fcff"
server_key_version: "encryptionkey@0"
I20260929 18:41:26.186285 32578 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707036041078-13712-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707036041078-13712-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707036041078-13712-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:41:26.207530 32578 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:41:26.208197 32578 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:41:26.208395 32578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:41:26.208973 32578 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:41:26.209872 32578 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:41:26.209962 32578 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:41:26.210034 32578 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:41:26.210088 32578 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:41:26.219321 32578 rpc_server.cc:307] RPC server started. Bound to: 127.13.100.2:33861
I20260929 18:41:26.219350 32705 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.100.2:33861 every 8 connection(s)
I20260929 18:41:26.220148 32578 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707036041078-13712-0/minicluster-data/ts-1/data/info.pb
I20260929 18:41:26.227867 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 32578
I20260929 18:41:26.228006 13712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707036041078-13712-0/minicluster-data/ts-1/wal/instance
I20260929 18:41:26.228315 13712 external_mini_cluster.cc:1428] Setting key 60de72b004ec521012022edab183d26b
I20260929 18:41:26.229180 32426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:26.221851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.2:59337 (local address 127.13.100.62:33573)
0929 18:41:26.222054 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0929 18:41:26.222058 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:41:26.223021 (+ 963us) server_negotiation.cc:407] Connection header received
0929 18:41:26.224339 (+ 1318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:41:26.224342 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:41:26.224386 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:41:26.224458 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:41:26.225343 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:26.225857 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:26.227069 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:26.227279 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:26.227352 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:26.228626 (+ 1274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:41:26.228630 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:41:26.228633 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:41:26.228647 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:41:26.228678 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:41:26.228688 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:41:26.228777 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0929 18:41:26.228955 (+ 178us) server_negotiation.cc:299] Negotiation successful
0929 18:41:26.229015 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260929 18:41:26.230070 32708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:26.222240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:33573 (local address 127.13.100.2:59337)
0929 18:41:26.222871 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:26.222910 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 18:41:26.224142 (+ 1232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:41:26.224658 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:41:26.224664 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:41:26.224741 (+ 77us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:41:26.225203 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:41:26.225215 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:41:26.226028 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:41:26.226031 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:41:26.226960 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:41:26.226969 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:41:26.227410 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:41:26.227412 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:41:26.227584 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:26.228317 (+ 733us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:41:26.228318 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:41:26.228355 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:41:26.228450 (+ 95us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:41:26.228778 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:41:26.228790 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:41:26.228799 (+ 9us) client_negotiation.cc:769] Sending connection context
0929 18:41:26.228953 (+ 154us) client_negotiation.cc:240] Negotiation successful
0929 18:41:26.229207 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":102,"threads_started":1}
I20260929 18:41:26.230154 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707036041078-13712-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707036041078-13712-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707036041078-13712-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707036041078-13712-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.100.3:0
--local_ip_for_outbound_sockets=127.13.100.3
--webserver_interface=127.13.100.3
--webserver_port=0
--tserver_master_addrs=127.13.100.62:33573
--builtin_ntp_servers=127.13.100.20:40615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 18:41:26.231204 32706 heartbeater.cc:347] Connected to a master server at 127.13.100.62:33573
I20260929 18:41:26.231446 32706 heartbeater.cc:464] Registering TS with master...
I20260929 18:41:26.231945 32706 heartbeater.cc:511] Master 127.13.100.62:33573 requested a full tablet report, sending...
I20260929 18:41:26.232987 32387 ts_manager.cc:194] Registered new tserver with Master: ad2825ca8afe48fb80046d88df4dd9da (127.13.100.2:33861)
I20260929 18:41:26.233584 32387 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.100.2:59337
W20260929 18:41:26.337138 32709 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:41:26.337374 32709 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:41:26.337411 32709 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:41:26.341526 32709 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:41:26.341575 32709 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:41:26.341655 32709 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.100.3
I20260929 18:41:26.346603 32709 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.100.20:40615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707036041078-13712-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707036041078-13712-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.100.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707036041078-13712-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.100.3
--webserver_port=0
--tserver_master_addrs=127.13.100.62:33573
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32709
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.100.3
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707036041078-13712-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:41:26.347570 32709 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:41:26.348862 32709 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 18:41:26.358336 32709 server_base.cc:1034] running on GCE node
W20260929 18:41:26.358330 32715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:41:26.358330 32717 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:41:26.358510 32714 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:41:26.358958 32709 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:41:26.359627 32709 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:41:26.360821 32709 hybrid_clock.cc:648] HybridClock initialized: now 1790707286360789 us; error 43 us; skew 500 ppm
I20260929 18:41:26.362924 32709 webserver.cc:460] Webserver started at http://127.13.100.3:42723/ using document root <none> and password file <none>
I20260929 18:41:26.363547 32709 fs_manager.cc:359] Metadata directory not provided
I20260929 18:41:26.363642 32709 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:41:26.363857 32709 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:41:26.365952 32709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707036041078-13712-0/minicluster-data/ts-2/data/instance:
uuid: "f11ad477477e4bf19520a869f6da2a89"
format_stamp: "Formatted at 2026-09-29 18:41:26 on dist-test-slave-j1xs"
server_key: "2116f244036aa332f25fa89f561b15e8"
server_key_iv: "2b6aacf16657f08acb218321e93bb6f7"
server_key_version: "encryptionkey@0"
I20260929 18:41:26.366549 32709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707036041078-13712-0/minicluster-data/ts-2/wal/instance:
uuid: "f11ad477477e4bf19520a869f6da2a89"
format_stamp: "Formatted at 2026-09-29 18:41:26 on dist-test-slave-j1xs"
server_key: "2116f244036aa332f25fa89f561b15e8"
server_key_iv: "2b6aacf16657f08acb218321e93bb6f7"
server_key_version: "encryptionkey@0"
I20260929 18:41:26.370129 32709 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:41:26.372454 32723 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:41:26.373632 32709 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:41:26.373762 32709 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707036041078-13712-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707036041078-13712-0/minicluster-data/ts-2/wal
uuid: "f11ad477477e4bf19520a869f6da2a89"
format_stamp: "Formatted at 2026-09-29 18:41:26 on dist-test-slave-j1xs"
server_key: "2116f244036aa332f25fa89f561b15e8"
server_key_iv: "2b6aacf16657f08acb218321e93bb6f7"
server_key_version: "encryptionkey@0"
I20260929 18:41:26.373871 32709 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707036041078-13712-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707036041078-13712-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707036041078-13712-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:41:26.392319 32709 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:41:26.392980 32709 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:41:26.393183 32709 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:41:26.393744 32709 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:41:26.394704 32709 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:41:26.394776 32709 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:41:26.394847 32709 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:41:26.394896 32709 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:41:26.403578 32709 rpc_server.cc:307] RPC server started. Bound to: 127.13.100.3:38603
I20260929 18:41:26.403610 368 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.100.3:38603 every 8 connection(s)
I20260929 18:41:26.404462 32709 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707036041078-13712-0/minicluster-data/ts-2/data/info.pb
I20260929 18:41:26.406425 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 32709
I20260929 18:41:26.406543 13712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707036041078-13712-0/minicluster-data/ts-2/wal/instance
I20260929 18:41:26.406874 13712 external_mini_cluster.cc:1428] Setting key 0b3cd86e29408918d87582b57c313fc2
I20260929 18:41:26.414201 32426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:26.406330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.3:50709 (local address 127.13.100.62:33573)
0929 18:41:26.406467 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0929 18:41:26.406470 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:41:26.407795 (+ 1325us) server_negotiation.cc:407] Connection header received
0929 18:41:26.409506 (+ 1711us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:41:26.409509 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:41:26.409563 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:41:26.409645 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:41:26.410527 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:26.411069 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:26.412208 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:26.412433 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:26.412539 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:26.413652 (+ 1113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:41:26.413655 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:41:26.413658 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:41:26.413671 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:41:26.413698 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:41:26.413705 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:41:26.413782 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 18:41:26.413997 (+ 215us) server_negotiation.cc:299] Negotiation successful
0929 18:41:26.414058 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 18:41:26.414996 371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:26.406870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:33573 (local address 127.13.100.3:50709)
0929 18:41:26.407663 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:26.407715 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 18:41:26.409305 (+ 1590us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:41:26.409808 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:41:26.409815 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:41:26.409914 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:41:26.410384 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:41:26.410396 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:41:26.411200 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:41:26.411204 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:41:26.412080 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:41:26.412089 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:41:26.412589 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:41:26.412593 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:41:26.412761 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:26.413391 (+ 630us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:41:26.413393 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:41:26.413431 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:41:26.413492 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:41:26.413796 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:41:26.413807 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:41:26.413813 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 18:41:26.413991 (+ 178us) client_negotiation.cc:240] Negotiation successful
0929 18:41:26.414271 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":130,"threads_started":1}
I20260929 18:41:26.416095 369 heartbeater.cc:347] Connected to a master server at 127.13.100.62:33573
I20260929 18:41:26.416306 369 heartbeater.cc:464] Registering TS with master...
I20260929 18:41:26.416769 369 heartbeater.cc:511] Master 127.13.100.62:33573 requested a full tablet report, sending...
I20260929 18:41:26.417811 32387 ts_manager.cc:194] Registered new tserver with Master: f11ad477477e4bf19520a869f6da2a89 (127.13.100.3:38603)
I20260929 18:41:26.418460 32387 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.100.3:50709
I20260929 18:41:26.421087 13712 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 18:41:26.427318 32426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:26.423279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54658 (local address 127.13.100.62:33573)
0929 18:41:26.423543 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0929 18:41:26.423547 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:41:26.423560 (+ 13us) server_negotiation.cc:407] Connection header received
0929 18:41:26.423707 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:41:26.423709 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:41:26.423748 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:41:26.423815 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:41:26.424081 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:26.424585 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:26.425509 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:26.425728 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:26.425808 (+ 80us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:26.426784 (+ 976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:41:26.426787 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:41:26.426789 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:41:26.426802 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:41:26.426819 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:41:26.426827 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:41:26.426926 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0929 18:41:26.427103 (+ 177us) server_negotiation.cc:299] Negotiation successful
0929 18:41:26.427163 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179}
I20260929 18:41:26.430655 32387 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54658:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:41:26.433585 32387 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:41:26.448400 381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:26.442702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.2:33861 (local address 127.0.0.1:58258)
0929 18:41:26.443608 (+ 906us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:26.443636 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0929 18:41:26.443722 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:41:26.444215 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:41:26.444220 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:41:26.444242 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:41:26.444400 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:41:26.444407 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:41:26.445559 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:41:26.445563 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:41:26.447045 (+ 1482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:41:26.447055 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:41:26.447847 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:41:26.447851 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:41:26.447959 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:26.447995 (+ 36us) client_negotiation.cc:769] Sending connection context
0929 18:41:26.448119 (+ 124us) client_negotiation.cc:240] Negotiation successful
0929 18:41:26.448235 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":853,"mutex_wait_us":65,"thread_start_us":95,"threads_started":1}
I20260929 18:41:26.448657 382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:26.442507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52256 (local address 127.13.100.1:33277)
0929 18:41:26.443683 (+ 1176us) server_negotiation.cc:206] Beginning negotiation
0929 18:41:26.443697 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 18:41:26.443719 (+ 22us) server_negotiation.cc:407] Connection header received
0929 18:41:26.443996 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:41:26.444011 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:41:26.444224 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:41:26.444528 (+ 304us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:41:26.444739 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:26.445863 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:26.447233 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:26.448019 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:26.448132 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:26.448340 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0929 18:41:26.448392 (+ 52us) server_negotiation.cc:299] Negotiation successful
0929 18:41:26.448496 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1066,"thread_start_us":110,"threads_started":1}
I20260929 18:41:26.448654 380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:26.442445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.3:38603 (local address 127.0.0.1:57254)
0929 18:41:26.443214 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:26.443264 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 18:41:26.443393 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:41:26.444352 (+ 959us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:41:26.444357 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:41:26.444379 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:41:26.444574 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:41:26.444582 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:41:26.445995 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:41:26.445999 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:41:26.447248 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:41:26.447256 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:41:26.448283 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:41:26.448286 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:41:26.448377 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:26.448400 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 18:41:26.448448 (+ 48us) client_negotiation.cc:240] Negotiation successful
0929 18:41:26.448510 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":664,"mutex_wait_us":78,"spinlock_wait_cycles":14464,"thread_start_us":96,"threads_started":1}
I20260929 18:41:26.448747 384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:26.443159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58258 (local address 127.13.100.2:33861)
0929 18:41:26.443626 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0929 18:41:26.443639 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 18:41:26.443697 (+ 58us) server_negotiation.cc:407] Connection header received
0929 18:41:26.443825 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:41:26.443831 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:41:26.444034 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:41:26.444551 (+ 517us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:41:26.444685 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:26.445439 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:26.447190 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:26.447723 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:26.447851 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:26.448046 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0929 18:41:26.448484 (+ 438us) server_negotiation.cc:299] Negotiation successful
0929 18:41:26.448591 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":112,"threads_started":1}
I20260929 18:41:26.448752 383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:26.442390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.1:33277 (local address 127.0.0.1:52256)
0929 18:41:26.443215 (+ 825us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:26.443263 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 18:41:26.443395 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:41:26.444385 (+ 990us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:41:26.444388 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:41:26.444403 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:41:26.444635 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:41:26.444645 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:41:26.446026 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:41:26.446031 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:41:26.447069 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:41:26.447078 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:41:26.448126 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:41:26.448130 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:41:26.448218 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:26.448238 (+ 20us) client_negotiation.cc:769] Sending connection context
0929 18:41:26.448296 (+ 58us) client_negotiation.cc:240] Negotiation successful
0929 18:41:26.448349 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":642,"mutex_wait_us":102,"spinlock_wait_cycles":6400,"thread_start_us":94,"threads_started":1}
I20260929 18:41:26.450768 385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:26.443221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57254 (local address 127.13.100.3:38603)
0929 18:41:26.443956 (+ 735us) server_negotiation.cc:206] Beginning negotiation
0929 18:41:26.443967 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 18:41:26.443986 (+ 19us) server_negotiation.cc:407] Connection header received
0929 18:41:26.444054 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:41:26.444059 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:41:26.444219 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:41:26.444359 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:41:26.444696 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:26.445879 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:26.447395 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:26.448154 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:26.450151 (+ 1997us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:26.450333 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0929 18:41:26.450523 (+ 190us) server_negotiation.cc:299] Negotiation successful
0929 18:41:26.450618 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":647,"thread_start_us":156,"threads_started":1}
I20260929 18:41:26.451845 32509 tablet_service.cc:1467] Processing CreateTablet for tablet e52139bdc71841cb95822c7273c40bea (DEFAULT_TABLE table=test-table [id=a507be2c4af5483c999e42137b0468d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:41:26.452047 32640 tablet_service.cc:1467] Processing CreateTablet for tablet e52139bdc71841cb95822c7273c40bea (DEFAULT_TABLE table=test-table [id=a507be2c4af5483c999e42137b0468d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:41:26.452847 32509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e52139bdc71841cb95822c7273c40bea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:41:26.453027 32640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e52139bdc71841cb95822c7273c40bea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:41:26.453267 303 tablet_service.cc:1467] Processing CreateTablet for tablet e52139bdc71841cb95822c7273c40bea (DEFAULT_TABLE table=test-table [id=a507be2c4af5483c999e42137b0468d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:41:26.454206 303 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e52139bdc71841cb95822c7273c40bea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:41:26.459631 386 tablet_bootstrap.cc:492] T e52139bdc71841cb95822c7273c40bea P f28cf2df86f0404c838a57df686e4634: Bootstrap starting.
I20260929 18:41:26.460613 387 tablet_bootstrap.cc:492] T e52139bdc71841cb95822c7273c40bea P ad2825ca8afe48fb80046d88df4dd9da: Bootstrap starting.
I20260929 18:41:26.461761 386 tablet_bootstrap.cc:654] T e52139bdc71841cb95822c7273c40bea P f28cf2df86f0404c838a57df686e4634: Neither blocks nor log segments found. Creating new log.
I20260929 18:41:26.462225 388 tablet_bootstrap.cc:492] T e52139bdc71841cb95822c7273c40bea P f11ad477477e4bf19520a869f6da2a89: Bootstrap starting.
I20260929 18:41:26.462433 387 tablet_bootstrap.cc:654] T e52139bdc71841cb95822c7273c40bea P ad2825ca8afe48fb80046d88df4dd9da: Neither blocks nor log segments found. Creating new log.
I20260929 18:41:26.462805 386 log.cc:824] T e52139bdc71841cb95822c7273c40bea P f28cf2df86f0404c838a57df686e4634: Log is configured to *not* fsync() on all Append() calls
I20260929 18:41:26.463251 387 log.cc:824] T e52139bdc71841cb95822c7273c40bea P ad2825ca8afe48fb80046d88df4dd9da: Log is configured to *not* fsync() on all Append() calls
I20260929 18:41:26.464447 388 tablet_bootstrap.cc:654] T e52139bdc71841cb95822c7273c40bea P f11ad477477e4bf19520a869f6da2a89: Neither blocks nor log segments found. Creating new log.
I20260929 18:41:26.464995 387 tablet_bootstrap.cc:492] T e52139bdc71841cb95822c7273c40bea P ad2825ca8afe48fb80046d88df4dd9da: No bootstrap required, opened a new log
I20260929 18:41:26.465210 387 ts_tablet_manager.cc:1397] T e52139bdc71841cb95822c7273c40bea P ad2825ca8afe48fb80046d88df4dd9da: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260929 18:41:26.465215 388 log.cc:824] T e52139bdc71841cb95822c7273c40bea P f11ad477477e4bf19520a869f6da2a89: Log is configured to *not* fsync() on all Append() calls
I20260929 18:41:26.465530 386 tablet_bootstrap.cc:492] T e52139bdc71841cb95822c7273c40bea P f28cf2df86f0404c838a57df686e4634: No bootstrap required, opened a new log
I20260929 18:41:26.465734 386 ts_tablet_manager.cc:1397] T e52139bdc71841cb95822c7273c40bea P f28cf2df86f0404c838a57df686e4634: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260929 18:41:26.466822 388 tablet_bootstrap.cc:492] T e52139bdc71841cb95822c7273c40bea P f11ad477477e4bf19520a869f6da2a89: No bootstrap required, opened a new log
I20260929 18:41:26.467010 388 ts_tablet_manager.cc:1397] T e52139bdc71841cb95822c7273c40bea P f11ad477477e4bf19520a869f6da2a89: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260929 18:41:26.469556 386 raft_consensus.cc:348] T e52139bdc71841cb95822c7273c40bea P f28cf2df86f0404c838a57df686e4634 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f28cf2df86f0404c838a57df686e4634" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 33277 } } peers { permanent_uuid: "f11ad477477e4bf19520a869f6da2a89" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 38603 } } peers { permanent_uuid: "ad2825ca8afe48fb80046d88df4dd9da" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 33861 } }
I20260929 18:41:26.469556 387 raft_consensus.cc:348] T e52139bdc71841cb95822c7273c40bea P ad2825ca8afe48fb80046d88df4dd9da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f28cf2df86f0404c838a57df686e4634" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 33277 } } peers { permanent_uuid: "f11ad477477e4bf19520a869f6da2a89" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 38603 } } peers { permanent_uuid: "ad2825ca8afe48fb80046d88df4dd9da" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 33861 } }
I20260929 18:41:26.469828 386 raft_consensus.cc:374] T e52139bdc71841cb95822c7273c40bea P f28cf2df86f0404c838a57df686e4634 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:41:26.469828 387 raft_consensus.cc:374] T e52139bdc71841cb95822c7273c40bea P ad2825ca8afe48fb80046d88df4dd9da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:41:26.469913 387 raft_consensus.cc:729] T e52139bdc71841cb95822c7273c40bea P ad2825ca8afe48fb80046d88df4dd9da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad2825ca8afe48fb80046d88df4dd9da, State: Initialized, Role: FOLLOWER
I20260929 18:41:26.469914 386 raft_consensus.cc:729] T e52139bdc71841cb95822c7273c40bea P f28cf2df86f0404c838a57df686e4634 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f28cf2df86f0404c838a57df686e4634, State: Initialized, Role: FOLLOWER
I20260929 18:41:26.469991 388 raft_consensus.cc:348] T e52139bdc71841cb95822c7273c40bea P f11ad477477e4bf19520a869f6da2a89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f28cf2df86f0404c838a57df686e4634" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 33277 } } peers { permanent_uuid: "f11ad477477e4bf19520a869f6da2a89" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 38603 } } peers { permanent_uuid: "ad2825ca8afe48fb80046d88df4dd9da" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 33861 } }
I20260929 18:41:26.470240 388 raft_consensus.cc:374] T e52139bdc71841cb95822c7273c40bea P f11ad477477e4bf19520a869f6da2a89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:41:26.470301 388 raft_consensus.cc:729] T e52139bdc71841cb95822c7273c40bea P f11ad477477e4bf19520a869f6da2a89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f11ad477477e4bf19520a869f6da2a89, State: Initialized, Role: FOLLOWER
I20260929 18:41:26.470394 387 consensus_queue.cc:260] T e52139bdc71841cb95822c7273c40bea P ad2825ca8afe48fb80046d88df4dd9da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f28cf2df86f0404c838a57df686e4634" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 33277 } } peers { permanent_uuid: "f11ad477477e4bf19520a869f6da2a89" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 38603 } } peers { permanent_uuid: "ad2825ca8afe48fb80046d88df4dd9da" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 33861 } }
I20260929 18:41:26.470394 386 consensus_queue.cc:260] T e52139bdc71841cb95822c7273c40bea P f28cf2df86f0404c838a57df686e4634 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f28cf2df86f0404c838a57df686e4634" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 33277 } } peers { permanent_uuid: "f11ad477477e4bf19520a869f6da2a89" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 38603 } } peers { permanent_uuid: "ad2825ca8afe48fb80046d88df4dd9da" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 33861 } }
I20260929 18:41:26.470748 388 consensus_queue.cc:260] T e52139bdc71841cb95822c7273c40bea P f11ad477477e4bf19520a869f6da2a89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f28cf2df86f0404c838a57df686e4634" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 33277 } } peers { permanent_uuid: "f11ad477477e4bf19520a869f6da2a89" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 38603 } } peers { permanent_uuid: "ad2825ca8afe48fb80046d88df4dd9da" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 33861 } }
I20260929 18:41:26.471422 32706 heartbeater.cc:503] Master 127.13.100.62:33573 was elected leader, sending a full tablet report...
I20260929 18:41:26.471536 32575 heartbeater.cc:503] Master 127.13.100.62:33573 was elected leader, sending a full tablet report...
I20260929 18:41:26.471558 386 ts_tablet_manager.cc:1428] T e52139bdc71841cb95822c7273c40bea P f28cf2df86f0404c838a57df686e4634: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260929 18:41:26.472317 369 heartbeater.cc:503] Master 127.13.100.62:33573 was elected leader, sending a full tablet report...
I20260929 18:41:26.472846 388 ts_tablet_manager.cc:1428] T e52139bdc71841cb95822c7273c40bea P f11ad477477e4bf19520a869f6da2a89: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260929 18:41:26.471407 387 ts_tablet_manager.cc:1428] T e52139bdc71841cb95822c7273c40bea P ad2825ca8afe48fb80046d88df4dd9da: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260929 18:41:26.502774 392 raft_consensus.cc:482] T e52139bdc71841cb95822c7273c40bea P f28cf2df86f0404c838a57df686e4634 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:41:26.502974 392 raft_consensus.cc:504] T e52139bdc71841cb95822c7273c40bea P f28cf2df86f0404c838a57df686e4634 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f28cf2df86f0404c838a57df686e4634" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 33277 } } peers { permanent_uuid: "f11ad477477e4bf19520a869f6da2a89" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 38603 } } peers { permanent_uuid: "ad2825ca8afe48fb80046d88df4dd9da" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 33861 } }
I20260929 18:41:26.504199 392 leader_election.cc:288] T e52139bdc71841cb95822c7273c40bea P f28cf2df86f0404c838a57df686e4634 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f11ad477477e4bf19520a869f6da2a89 (127.13.100.3:38603), ad2825ca8afe48fb80046d88df4dd9da (127.13.100.2:33861)
I20260929 18:41:26.508487 32577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:26.504408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.3:38603 (local address 127.13.100.1:54457)
0929 18:41:26.504597 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:26.504615 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 18:41:26.504720 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:41:26.505078 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:41:26.505081 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:41:26.505104 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:41:26.505239 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:41:26.505247 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:41:26.506318 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:41:26.506322 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:41:26.507188 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:41:26.507197 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:41:26.508053 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:41:26.508056 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:41:26.508156 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:26.508182 (+ 26us) client_negotiation.cc:769] Sending connection context
0929 18:41:26.508261 (+ 79us) client_negotiation.cc:240] Negotiation successful
0929 18:41:26.508345 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106}
I20260929 18:41:26.508558 385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:26.504481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.1:54457 (local address 127.13.100.3:38603)
0929 18:41:26.504603 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0929 18:41:26.504607 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:41:26.504824 (+ 217us) server_negotiation.cc:407] Connection header received
0929 18:41:26.504890 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:41:26.504893 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:41:26.504950 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:41:26.505068 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:41:26.505376 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:26.506205 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:26.507347 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:26.507936 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:26.508031 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:26.508079 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0929 18:41:26.508349 (+ 270us) server_negotiation.cc:299] Negotiation successful
0929 18:41:26.508431 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 18:41:26.509092 384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:26.504816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.1:47425 (local address 127.13.100.2:33861)
0929 18:41:26.504986 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0929 18:41:26.504993 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 18:41:26.505276 (+ 283us) server_negotiation.cc:407] Connection header received
0929 18:41:26.505434 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:41:26.505437 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:41:26.505479 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:41:26.505536 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:41:26.505850 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:26.506498 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:26.507778 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:26.508264 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:26.508364 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:26.508405 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0929 18:41:26.508902 (+ 497us) server_negotiation.cc:299] Negotiation successful
0929 18:41:26.508980 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 18:41:26.509159 395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:26.504645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.2:33861 (local address 127.13.100.1:47425)
0929 18:41:26.505196 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:26.505215 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 18:41:26.505293 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:41:26.505584 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:41:26.505587 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:41:26.505605 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:41:26.505736 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:41:26.505743 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:41:26.506636 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:41:26.506638 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:41:26.507648 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:41:26.507659 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:41:26.508404 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:41:26.508408 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:41:26.508503 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:26.508524 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 18:41:26.508900 (+ 376us) client_negotiation.cc:240] Negotiation successful
0929 18:41:26.508992 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":92,"threads_started":1}
I20260929 18:41:26.509236 323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e52139bdc71841cb95822c7273c40bea" candidate_uuid: "f28cf2df86f0404c838a57df686e4634" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f11ad477477e4bf19520a869f6da2a89" is_pre_election: true
I20260929 18:41:26.509583 323 raft_consensus.cc:2454] T e52139bdc71841cb95822c7273c40bea P f11ad477477e4bf19520a869f6da2a89 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f28cf2df86f0404c838a57df686e4634 in term 0.
I20260929 18:41:26.509776 32660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e52139bdc71841cb95822c7273c40bea" candidate_uuid: "f28cf2df86f0404c838a57df686e4634" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad2825ca8afe48fb80046d88df4dd9da" is_pre_election: true
I20260929 18:41:26.510074 32660 raft_consensus.cc:2454] T e52139bdc71841cb95822c7273c40bea P ad2825ca8afe48fb80046d88df4dd9da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f28cf2df86f0404c838a57df686e4634 in term 0.
I20260929 18:41:26.510105 32462 leader_election.cc:302] T e52139bdc71841cb95822c7273c40bea P f28cf2df86f0404c838a57df686e4634 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f11ad477477e4bf19520a869f6da2a89, f28cf2df86f0404c838a57df686e4634; no voters:
I20260929 18:41:26.510425 392 raft_consensus.cc:2781] T e52139bdc71841cb95822c7273c40bea P f28cf2df86f0404c838a57df686e4634 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:41:26.510499 392 raft_consensus.cc:482] T e52139bdc71841cb95822c7273c40bea P f28cf2df86f0404c838a57df686e4634 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:41:26.510528 392 raft_consensus.cc:3037] T e52139bdc71841cb95822c7273c40bea P f28cf2df86f0404c838a57df686e4634 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:41:26.511560 392 raft_consensus.cc:504] T e52139bdc71841cb95822c7273c40bea P f28cf2df86f0404c838a57df686e4634 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f28cf2df86f0404c838a57df686e4634" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 33277 } } peers { permanent_uuid: "f11ad477477e4bf19520a869f6da2a89" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 38603 } } peers { permanent_uuid: "ad2825ca8afe48fb80046d88df4dd9da" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 33861 } }
I20260929 18:41:26.511919 392 leader_election.cc:288] T e52139bdc71841cb95822c7273c40bea P f28cf2df86f0404c838a57df686e4634 [CANDIDATE]: Term 1 election: Requested vote from peers f11ad477477e4bf19520a869f6da2a89 (127.13.100.3:38603), ad2825ca8afe48fb80046d88df4dd9da (127.13.100.2:33861)
I20260929 18:41:26.512372 323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e52139bdc71841cb95822c7273c40bea" candidate_uuid: "f28cf2df86f0404c838a57df686e4634" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f11ad477477e4bf19520a869f6da2a89"
I20260929 18:41:26.512476 32660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e52139bdc71841cb95822c7273c40bea" candidate_uuid: "f28cf2df86f0404c838a57df686e4634" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad2825ca8afe48fb80046d88df4dd9da"
I20260929 18:41:26.512483 323 raft_consensus.cc:3037] T e52139bdc71841cb95822c7273c40bea P f11ad477477e4bf19520a869f6da2a89 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:41:26.512648 32660 raft_consensus.cc:3037] T e52139bdc71841cb95822c7273c40bea P ad2825ca8afe48fb80046d88df4dd9da [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:41:26.513604 323 raft_consensus.cc:2454] T e52139bdc71841cb95822c7273c40bea P f11ad477477e4bf19520a869f6da2a89 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f28cf2df86f0404c838a57df686e4634 in term 1.
I20260929 18:41:26.513605 32660 raft_consensus.cc:2454] T e52139bdc71841cb95822c7273c40bea P ad2825ca8afe48fb80046d88df4dd9da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f28cf2df86f0404c838a57df686e4634 in term 1.
I20260929 18:41:26.513990 32465 leader_election.cc:302] T e52139bdc71841cb95822c7273c40bea P f28cf2df86f0404c838a57df686e4634 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad2825ca8afe48fb80046d88df4dd9da, f28cf2df86f0404c838a57df686e4634; no voters:
I20260929 18:41:26.514204 392 raft_consensus.cc:2781] T e52139bdc71841cb95822c7273c40bea P f28cf2df86f0404c838a57df686e4634 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:41:26.514390 392 raft_consensus.cc:686] T e52139bdc71841cb95822c7273c40bea P f28cf2df86f0404c838a57df686e4634 [term 1 LEADER]: Becoming Leader. State: Replica: f28cf2df86f0404c838a57df686e4634, State: Running, Role: LEADER
I20260929 18:41:26.514722 392 consensus_queue.cc:237] T e52139bdc71841cb95822c7273c40bea P f28cf2df86f0404c838a57df686e4634 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f28cf2df86f0404c838a57df686e4634" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 33277 } } peers { permanent_uuid: "f11ad477477e4bf19520a869f6da2a89" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 38603 } } peers { permanent_uuid: "ad2825ca8afe48fb80046d88df4dd9da" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 33861 } }
I20260929 18:41:26.517807 32386 catalog_manager.cc:5495] T e52139bdc71841cb95822c7273c40bea P f28cf2df86f0404c838a57df686e4634 reported cstate change: term changed from 0 to 1, leader changed from <none> to f28cf2df86f0404c838a57df686e4634 (127.13.100.1). New cstate: current_term: 1 leader_uuid: "f28cf2df86f0404c838a57df686e4634" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f28cf2df86f0404c838a57df686e4634" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 33277 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f11ad477477e4bf19520a869f6da2a89" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 38603 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad2825ca8afe48fb80046d88df4dd9da" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 33861 } health_report { overall_health: UNKNOWN } } }
I20260929 18:41:26.529309 382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:26.525798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52270 (local address 127.13.100.1:33277)
0929 18:41:26.525960 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0929 18:41:26.525965 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:41:26.525979 (+ 14us) server_negotiation.cc:407] Connection header received
0929 18:41:26.526149 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:41:26.526152 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:41:26.526208 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:41:26.526301 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:41:26.526521 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:26.527029 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:26.527880 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:26.528106 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:26.528185 (+ 79us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:26.528476 (+ 291us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:41:26.528864 (+ 388us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:41:26.528972 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 18:41:26.529150 (+ 178us) server_negotiation.cc:299] Negotiation successful
0929 18:41:26.529182 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260929 18:41:26.531754 32576 tablet.cc:2369] T e52139bdc71841cb95822c7273c40bea P f28cf2df86f0404c838a57df686e4634: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:41:26.534956 323 raft_consensus.cc:1261] T e52139bdc71841cb95822c7273c40bea P f11ad477477e4bf19520a869f6da2a89 [term 1 FOLLOWER]: Refusing update from remote peer f28cf2df86f0404c838a57df686e4634: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:41:26.534957 32660 raft_consensus.cc:1261] T e52139bdc71841cb95822c7273c40bea P ad2825ca8afe48fb80046d88df4dd9da [term 1 FOLLOWER]: Refusing update from remote peer f28cf2df86f0404c838a57df686e4634: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:41:26.535626 392 consensus_queue.cc:1035] T e52139bdc71841cb95822c7273c40bea P f28cf2df86f0404c838a57df686e4634 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f11ad477477e4bf19520a869f6da2a89" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 38603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:41:26.535830 396 consensus_queue.cc:1035] T e52139bdc71841cb95822c7273c40bea P f28cf2df86f0404c838a57df686e4634 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad2825ca8afe48fb80046d88df4dd9da" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 33861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:41:26.543179 400 mvcc.cc:205] Tried to move back new op lower bound from 7334737045640617984 to 7334737045567168512. Current Snapshot: MvccSnapshot[applied={T|T < 7334737045640617984}]
I20260929 18:41:26.543318 399 mvcc.cc:205] Tried to move back new op lower bound from 7334737045640617984 to 7334737045567168512. Current Snapshot: MvccSnapshot[applied={T|T < 7334737045640617984}]
I20260929 18:41:26.551306 32387 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54658:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:41:26.551499 32387 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54658:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:41:26.553824 32387 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5e36295463d640bca4e3d6bfd02f9d84: Sending DeleteTablet for 3 replicas of tablet e52139bdc71841cb95822c7273c40bea
I20260929 18:41:26.554659 32509 tablet_service.cc:1514] Processing DeleteTablet for tablet e52139bdc71841cb95822c7273c40bea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:41:26 UTC) from {username='slave'} at 127.0.0.1:52256
I20260929 18:41:26.554659 303 tablet_service.cc:1514] Processing DeleteTablet for tablet e52139bdc71841cb95822c7273c40bea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:41:26 UTC) from {username='slave'} at 127.0.0.1:57254
I20260929 18:41:26.555174 408 tablet_replica.cc:330] T e52139bdc71841cb95822c7273c40bea P f28cf2df86f0404c838a57df686e4634: stopping tablet replica
I20260929 18:41:26.555613 32640 tablet_service.cc:1514] Processing DeleteTablet for tablet e52139bdc71841cb95822c7273c40bea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:41:26 UTC) from {username='slave'} at 127.0.0.1:58258
I20260929 18:41:26.556005 408 raft_consensus.cc:2229] T e52139bdc71841cb95822c7273c40bea P f28cf2df86f0404c838a57df686e4634 [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:41:26.556061 410 tablet_replica.cc:330] T e52139bdc71841cb95822c7273c40bea P ad2825ca8afe48fb80046d88df4dd9da: stopping tablet replica
I20260929 18:41:26.556197 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 32447
I20260929 18:41:26.556471 410 raft_consensus.cc:2229] T e52139bdc71841cb95822c7273c40bea P ad2825ca8afe48fb80046d88df4dd9da [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:41:26.556815 410 raft_consensus.cc:2258] T e52139bdc71841cb95822c7273c40bea P ad2825ca8afe48fb80046d88df4dd9da [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:41:26.558135 410 ts_tablet_manager.cc:1905] T e52139bdc71841cb95822c7273c40bea P ad2825ca8afe48fb80046d88df4dd9da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:41:26.560487 410 ts_tablet_manager.cc:1918] T e52139bdc71841cb95822c7273c40bea P ad2825ca8afe48fb80046d88df4dd9da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:41:26.560600 410 log.cc:1196] T e52139bdc71841cb95822c7273c40bea P ad2825ca8afe48fb80046d88df4dd9da: Deleting WAL directory at /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707036041078-13712-0/minicluster-data/ts-1/wal/wals/e52139bdc71841cb95822c7273c40bea
I20260929 18:41:26.560873 409 tablet_replica.cc:330] T e52139bdc71841cb95822c7273c40bea P f11ad477477e4bf19520a869f6da2a89: stopping tablet replica
I20260929 18:41:26.560963 410 ts_tablet_manager.cc:1939] T e52139bdc71841cb95822c7273c40bea P ad2825ca8afe48fb80046d88df4dd9da: Deleting consensus metadata
I20260929 18:41:26.561203 409 raft_consensus.cc:2229] T e52139bdc71841cb95822c7273c40bea P f11ad477477e4bf19520a869f6da2a89 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:41:26.561527 409 raft_consensus.cc:2258] T e52139bdc71841cb95822c7273c40bea P f11ad477477e4bf19520a869f6da2a89 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:41:26.562177 32374 catalog_manager.cc:4841] TS ad2825ca8afe48fb80046d88df4dd9da (127.13.100.2:33861): tablet e52139bdc71841cb95822c7273c40bea (table test-table [id=a507be2c4af5483c999e42137b0468d1]) successfully deleted
I20260929 18:41:26.563205 409 ts_tablet_manager.cc:1905] T e52139bdc71841cb95822c7273c40bea P f11ad477477e4bf19520a869f6da2a89: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:41:26.564747 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 32578
W20260929 18:41:26.564716 32372 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.100.1:33277 (error 108)
I20260929 18:41:26.566704 381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:26.566416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.1:33277 (local address 127.0.0.1:52272)
0929 18:41:26.566548 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:26.566635 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.1:33277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260929 18:41:26.566891 32372 catalog_manager.cc:4584] TS f28cf2df86f0404c838a57df686e4634 (127.13.100.1:33277): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e52139bdc71841cb95822c7273c40bea: Network error: Client connection negotiation failed: client connection to 127.13.100.1:33277: connect: Connection refused (error 111)
I20260929 18:41:26.567646 409 ts_tablet_manager.cc:1918] T e52139bdc71841cb95822c7273c40bea P f11ad477477e4bf19520a869f6da2a89: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:41:26.567756 409 log.cc:1196] T e52139bdc71841cb95822c7273c40bea P f11ad477477e4bf19520a869f6da2a89: Deleting WAL directory at /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707036041078-13712-0/minicluster-data/ts-2/wal/wals/e52139bdc71841cb95822c7273c40bea
I20260929 18:41:26.568111 409 ts_tablet_manager.cc:1939] T e52139bdc71841cb95822c7273c40bea P f11ad477477e4bf19520a869f6da2a89: Deleting consensus metadata
I20260929 18:41:26.569473 32371 catalog_manager.cc:4841] TS f11ad477477e4bf19520a869f6da2a89 (127.13.100.3:38603): tablet e52139bdc71841cb95822c7273c40bea (table test-table [id=a507be2c4af5483c999e42137b0468d1]) successfully deleted
I20260929 18:41:26.571449 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 32709
I20260929 18:41:26.577067 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 32356
2026-09-29T18:41:26Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (993 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707036041078-13712-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:41:26 dist-test-slave-j1xs krb5kdc[414](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:41:26 dist-test-slave-j1xs krb5kdc[414](info): set up 2 sockets
Sep 29 18:41:26 dist-test-slave-j1xs krb5kdc[414](info): commencing operation
krb5kdc: starting...
W20260929 18:41:27.613970 13712 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.007s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707036041078-13712-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707036041078-13712-0/krb5kdc/test-user.keytab.
Sep 29 18:41:27 dist-test-slave-j1xs krb5kdc[414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707287, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:41:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:41:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.100.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.100.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.100.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.62.keytab.
Entry for principal kudu/127.13.100.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.62.keytab.
I20260929 18:41:27.733417 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707036041078-13712-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707036041078-13712-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707036041078-13712-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.100.62:35021
--webserver_interface=127.13.100.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.62.keytab
--principal=kudu/127.13.100.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.100.20:34373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.100.62:35021
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707036041078-13712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707036041078-13712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707036041078-13712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:41:27.842262 427 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:41:27.842555 427 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:41:27.842644 427 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:41:27.846638 427 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:41:27.846726 427 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:41:27.846777 427 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:41:27.846843 427 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:41:27.846886 427 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:41:27.852072 427 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.100.20:34373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707036041078-13712-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707036041078-13712-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.100.62:35021
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.62.keytab
--principal=kudu/127.13.100.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.100.62:35021
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.100.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.427
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707036041078-13712-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:41:27.853083 427 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:41:27.854147 427 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:41:27.862466 432 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:41:27.862517 435 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:41:27.862489 433 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:41:27.862804 427 server_base.cc:1034] running on GCE node
I20260929 18:41:27.863348 427 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:41:27.864460 427 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:41:27.865679 427 hybrid_clock.cc:648] HybridClock initialized: now 1790707287865631 us; error 34 us; skew 500 ppm
Sep 29 18:41:27 dist-test-slave-j1xs krb5kdc[414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707287, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:41:27.868409 427 init.cc:377] Logged in from keytab as kudu/127.13.100.62@KRBTEST.COM (short username kudu)
I20260929 18:41:27.869278 427 webserver.cc:460] Webserver started at http://127.13.100.62:46881/ using document root <none> and password file <none>
I20260929 18:41:27.869825 427 fs_manager.cc:359] Metadata directory not provided
I20260929 18:41:27.869938 427 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:41:27.870174 427 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:41:27.872097 427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707036041078-13712-0/minicluster-data/master-0/data/instance:
uuid: "320d980920d94ca8bb71b32ae1729865"
format_stamp: "Formatted at 2026-09-29 18:41:27 on dist-test-slave-j1xs"
server_key: "34dd4603c5c6760bdba9cb60f624f130"
server_key_iv: "16b7ef062e5cc30b9931e0acbcb0ef2a"
server_key_version: "encryptionkey@0"
I20260929 18:41:27.872653 427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707036041078-13712-0/minicluster-data/master-0/wal/instance:
uuid: "320d980920d94ca8bb71b32ae1729865"
format_stamp: "Formatted at 2026-09-29 18:41:27 on dist-test-slave-j1xs"
server_key: "34dd4603c5c6760bdba9cb60f624f130"
server_key_iv: "16b7ef062e5cc30b9931e0acbcb0ef2a"
server_key_version: "encryptionkey@0"
I20260929 18:41:27.876250 427 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20260929 18:41:27.878681 442 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:41:27.879998 427 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260929 18:41:27.880103 427 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707036041078-13712-0/minicluster-data/master-0/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707036041078-13712-0/minicluster-data/master-0/wal
uuid: "320d980920d94ca8bb71b32ae1729865"
format_stamp: "Formatted at 2026-09-29 18:41:27 on dist-test-slave-j1xs"
server_key: "34dd4603c5c6760bdba9cb60f624f130"
server_key_iv: "16b7ef062e5cc30b9931e0acbcb0ef2a"
server_key_version: "encryptionkey@0"
I20260929 18:41:27.880189 427 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707036041078-13712-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707036041078-13712-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707036041078-13712-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:41:27.893613 427 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:41:27.896350 427 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:41:27.896499 427 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:41:27.903491 427 rpc_server.cc:307] RPC server started. Bound to: 127.13.100.62:35021
I20260929 18:41:27.903512 494 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.100.62:35021 every 8 connection(s)
I20260929 18:41:27.904390 427 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
I20260929 18:41:27.907051 495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:41:27.909447 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 427
I20260929 18:41:27.909595 13712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707036041078-13712-0/minicluster-data/master-0/wal/instance
I20260929 18:41:27.909945 13712 external_mini_cluster.cc:1428] Setting key 1ef76c29efec5c21f183e14adc0edb1a
I20260929 18:41:27.913033 495 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 320d980920d94ca8bb71b32ae1729865: Bootstrap starting.
I20260929 18:41:27.915303 495 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 320d980920d94ca8bb71b32ae1729865: Neither blocks nor log segments found. Creating new log.
I20260929 18:41:27.916020 495 log.cc:824] T 00000000000000000000000000000000 P 320d980920d94ca8bb71b32ae1729865: Log is configured to *not* fsync() on all Append() calls
I20260929 18:41:27.918004 495 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 320d980920d94ca8bb71b32ae1729865: No bootstrap required, opened a new log
Sep 29 18:41:27 dist-test-slave-j1xs krb5kdc[414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707287, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.100.62@KRBTEST.COM
I20260929 18:41:27.921326 495 raft_consensus.cc:348] T 00000000000000000000000000000000 P 320d980920d94ca8bb71b32ae1729865 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "320d980920d94ca8bb71b32ae1729865" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 35021 } }
I20260929 18:41:27.921569 495 raft_consensus.cc:374] T 00000000000000000000000000000000 P 320d980920d94ca8bb71b32ae1729865 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:41:27.921653 495 raft_consensus.cc:729] T 00000000000000000000000000000000 P 320d980920d94ca8bb71b32ae1729865 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 320d980920d94ca8bb71b32ae1729865, State: Initialized, Role: FOLLOWER
I20260929 18:41:27.922137 495 consensus_queue.cc:260] T 00000000000000000000000000000000 P 320d980920d94ca8bb71b32ae1729865 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "320d980920d94ca8bb71b32ae1729865" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 35021 } }
I20260929 18:41:27.922292 495 raft_consensus.cc:388] T 00000000000000000000000000000000 P 320d980920d94ca8bb71b32ae1729865 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:41:27.922363 495 raft_consensus.cc:482] T 00000000000000000000000000000000 P 320d980920d94ca8bb71b32ae1729865 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:41:27.922462 495 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 320d980920d94ca8bb71b32ae1729865 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:41:27.923327 495 raft_consensus.cc:504] T 00000000000000000000000000000000 P 320d980920d94ca8bb71b32ae1729865 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "320d980920d94ca8bb71b32ae1729865" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 35021 } }
I20260929 18:41:27.923676 495 leader_election.cc:302] T 00000000000000000000000000000000 P 320d980920d94ca8bb71b32ae1729865 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 320d980920d94ca8bb71b32ae1729865; no voters:
I20260929 18:41:27.924007 495 leader_election.cc:288] T 00000000000000000000000000000000 P 320d980920d94ca8bb71b32ae1729865 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:41:27.924080 500 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 320d980920d94ca8bb71b32ae1729865 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:41:27.924265 500 raft_consensus.cc:686] T 00000000000000000000000000000000 P 320d980920d94ca8bb71b32ae1729865 [term 1 LEADER]: Becoming Leader. State: Replica: 320d980920d94ca8bb71b32ae1729865, State: Running, Role: LEADER
I20260929 18:41:27.924582 498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:27.911460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41416 (local address 127.13.100.62:35021)
0929 18:41:27.911867 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0929 18:41:27.911881 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 18:41:27.911914 (+ 33us) server_negotiation.cc:407] Connection header received
0929 18:41:27.912509 (+ 595us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:41:27.912524 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:41:27.912842 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:41:27.913168 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:41:27.914015 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:27.915015 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:27.915924 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:27.916442 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:27.916628 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:27.919052 (+ 2424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:41:27.919058 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:41:27.919070 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:41:27.919096 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:41:27.921328 (+ 2232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:41:27.921827 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:41:27.921832 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:41:27.921836 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:41:27.922038 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:41:27.922398 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:41:27.922401 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:41:27.922406 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:41:27.922846 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:41:27.923056 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0929 18:41:27.923321 (+ 265us) server_negotiation.cc:299] Negotiation successful
0929 18:41:27.923632 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":114,"threads_started":1}
I20260929 18:41:27.925318 495 sys_catalog.cc:564] T 00000000000000000000000000000000 P 320d980920d94ca8bb71b32ae1729865 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:41:27.924563 500 consensus_queue.cc:237] T 00000000000000000000000000000000 P 320d980920d94ca8bb71b32ae1729865 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "320d980920d94ca8bb71b32ae1729865" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 35021 } }
I20260929 18:41:27.927699 501 sys_catalog.cc:455] T 00000000000000000000000000000000 P 320d980920d94ca8bb71b32ae1729865 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "320d980920d94ca8bb71b32ae1729865" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "320d980920d94ca8bb71b32ae1729865" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 35021 } } }
I20260929 18:41:27.927832 501 sys_catalog.cc:458] T 00000000000000000000000000000000 P 320d980920d94ca8bb71b32ae1729865 [sys.catalog]: This master's current role is: LEADER
I20260929 18:41:27.927847 502 sys_catalog.cc:455] T 00000000000000000000000000000000 P 320d980920d94ca8bb71b32ae1729865 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 320d980920d94ca8bb71b32ae1729865. Latest consensus state: current_term: 1 leader_uuid: "320d980920d94ca8bb71b32ae1729865" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "320d980920d94ca8bb71b32ae1729865" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 35021 } } }
I20260929 18:41:27.928009 502 sys_catalog.cc:458] T 00000000000000000000000000000000 P 320d980920d94ca8bb71b32ae1729865 [sys.catalog]: This master's current role is: LEADER
W20260929 18:41:27.930516 514 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 320d980920d94ca8bb71b32ae1729865: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:41:27.930578 514 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:41:27.930668 515 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:41:27.931349 515 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:41:27.936548 515 catalog_manager.cc:1347] Generated new cluster ID: 526970328b7948ceae16a793bd51765b
I20260929 18:41:27.936609 515 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:41:27.951874 515 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:41:27.953141 515 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:41:27.959739 515 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 320d980920d94ca8bb71b32ae1729865: Generated new TSK 0
I20260929 18:41:27.960528 515 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.100.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.100.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.100.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.1.keytab.
Entry for principal kudu/127.13.100.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.1.keytab.
I20260929 18:41:27.996520 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707036041078-13712-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707036041078-13712-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707036041078-13712-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707036041078-13712-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.100.1:0
--local_ip_for_outbound_sockets=127.13.100.1
--webserver_interface=127.13.100.1
--webserver_port=0
--tserver_master_addrs=127.13.100.62:35021
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.1.keytab
--principal=kudu/127.13.100.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.100.20:34373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707036041078-13712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707036041078-13712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707036041078-13712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:41:28.107393 521 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:41:28.107666 521 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:41:28.107717 521 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:41:28.111871 521 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:41:28.111930 521 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:41:28.112142 521 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.100.1
I20260929 18:41:28.117537 521 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.100.20:34373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707036041078-13712-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707036041078-13712-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.1.keytab
--principal=kudu/127.13.100.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.100.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707036041078-13712-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.100.1
--webserver_port=0
--tserver_master_addrs=127.13.100.62:35021
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.521
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.100.1
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707036041078-13712-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:41:28.118649 521 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:41:28.119593 521 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:41:28.129104 527 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:41:28.129149 529 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:41:28.129259 521 server_base.cc:1034] running on GCE node
W20260929 18:41:28.129184 526 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:41:28.129978 521 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:41:28.130679 521 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:41:28.131860 521 hybrid_clock.cc:648] HybridClock initialized: now 1790707288131855 us; error 64 us; skew 500 ppm
Sep 29 18:41:28 dist-test-slave-j1xs krb5kdc[414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707288, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:41:28.134827 521 init.cc:377] Logged in from keytab as kudu/127.13.100.1@KRBTEST.COM (short username kudu)
I20260929 18:41:28.135741 521 webserver.cc:460] Webserver started at http://127.13.100.1:36259/ using document root <none> and password file <none>
I20260929 18:41:28.136313 521 fs_manager.cc:359] Metadata directory not provided
I20260929 18:41:28.136404 521 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:41:28.136643 521 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:41:28.138790 521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707036041078-13712-0/minicluster-data/ts-0/data/instance:
uuid: "791a030d2d3f4d5ab760fd83cf19cdf5"
format_stamp: "Formatted at 2026-09-29 18:41:28 on dist-test-slave-j1xs"
server_key: "fd8897aa7377ff3758a6c003a466a090"
server_key_iv: "256b74270ef66e7a133e626247da1add"
server_key_version: "encryptionkey@0"
I20260929 18:41:28.139349 521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707036041078-13712-0/minicluster-data/ts-0/wal/instance:
uuid: "791a030d2d3f4d5ab760fd83cf19cdf5"
format_stamp: "Formatted at 2026-09-29 18:41:28 on dist-test-slave-j1xs"
server_key: "fd8897aa7377ff3758a6c003a466a090"
server_key_iv: "256b74270ef66e7a133e626247da1add"
server_key_version: "encryptionkey@0"
I20260929 18:41:28.143139 521 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 18:41:28.145572 536 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:41:28.146894 521 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:41:28.147038 521 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707036041078-13712-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707036041078-13712-0/minicluster-data/ts-0/wal
uuid: "791a030d2d3f4d5ab760fd83cf19cdf5"
format_stamp: "Formatted at 2026-09-29 18:41:28 on dist-test-slave-j1xs"
server_key: "fd8897aa7377ff3758a6c003a466a090"
server_key_iv: "256b74270ef66e7a133e626247da1add"
server_key_version: "encryptionkey@0"
I20260929 18:41:28.147152 521 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707036041078-13712-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707036041078-13712-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707036041078-13712-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:41:28.161644 521 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:41:28.164911 521 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:41:28.165164 521 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:41:28.165798 521 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:41:28.166651 521 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:41:28.166724 521 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:41:28.166808 521 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:41:28.166863 521 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:41:28.175747 521 rpc_server.cc:307] RPC server started. Bound to: 127.13.100.1:42395
I20260929 18:41:28.175812 649 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.100.1:42395 every 8 connection(s)
I20260929 18:41:28.176625 521 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707036041078-13712-0/minicluster-data/ts-0/data/info.pb
I20260929 18:41:28.183012 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 521
I20260929 18:41:28.183180 13712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707036041078-13712-0/minicluster-data/ts-0/wal/instance
I20260929 18:41:28.183482 13712 external_mini_cluster.cc:1428] Setting key d7a2bd80595dd51d728cea298e4c8aba
Sep 29 18:41:28 dist-test-slave-j1xs krb5kdc[414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707288, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.1@KRBTEST.COM for kudu/127.13.100.62@KRBTEST.COM
I20260929 18:41:28.190997 498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:28.178522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.1:38841 (local address 127.13.100.62:35021)
0929 18:41:28.178673 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0929 18:41:28.178678 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:41:28.179506 (+ 828us) server_negotiation.cc:407] Connection header received
0929 18:41:28.180883 (+ 1377us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:41:28.180886 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:41:28.180940 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:41:28.181031 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:41:28.182280 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:28.182806 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:28.184179 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:28.184410 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:28.184491 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:28.187243 (+ 2752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:41:28.187246 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:41:28.187248 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:41:28.187270 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:41:28.188875 (+ 1605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:41:28.189395 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:41:28.189400 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:41:28.189402 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:41:28.189486 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:41:28.189827 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:41:28.189831 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:41:28.189834 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:41:28.190071 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:41:28.190200 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0929 18:41:28.190699 (+ 499us) server_negotiation.cc:299] Negotiation successful
0929 18:41:28.190825 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 18:41:28.191802 652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:28.178908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:35021 (local address 127.13.100.1:38841)
0929 18:41:28.179353 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:28.179396 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 18:41:28.180682 (+ 1286us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:41:28.181222 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:41:28.181229 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:41:28.181560 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:41:28.182145 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:41:28.182162 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:41:28.182985 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:41:28.182987 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:41:28.184030 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:41:28.184050 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:41:28.184556 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:41:28.184560 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:41:28.184766 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:28.185351 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:41:28.185377 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:41:28.187057 (+ 1680us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:41:28.188990 (+ 1933us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:41:28.188996 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:41:28.189012 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:41:28.189290 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:41:28.189580 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:41:28.189584 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:41:28.189586 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:41:28.189731 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:41:28.190201 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:41:28.190209 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:41:28.190452 (+ 243us) client_negotiation.cc:769] Sending connection context
0929 18:41:28.190637 (+ 185us) client_negotiation.cc:240] Negotiation successful
0929 18:41:28.190957 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":102,"threads_started":1}
WARNING: no policy specified for kudu/127.13.100.2@KRBTEST.COM; defaulting to no policy
I20260929 18:41:28.193012 650 heartbeater.cc:347] Connected to a master server at 127.13.100.62:35021
I20260929 18:41:28.193231 650 heartbeater.cc:464] Registering TS with master...
I20260929 18:41:28.193725 650 heartbeater.cc:511] Master 127.13.100.62:35021 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.100.2@KRBTEST.COM" created.
I20260929 18:41:28.195250 459 ts_manager.cc:194] Registered new tserver with Master: 791a030d2d3f4d5ab760fd83cf19cdf5 (127.13.100.1:42395)
I20260929 18:41:28.196838 459 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.100.1@KRBTEST.COM'} at 127.13.100.1:38841
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.100.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.2.keytab.
Entry for principal kudu/127.13.100.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.2.keytab.
I20260929 18:41:28.208730 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707036041078-13712-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707036041078-13712-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707036041078-13712-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707036041078-13712-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.100.2:0
--local_ip_for_outbound_sockets=127.13.100.2
--webserver_interface=127.13.100.2
--webserver_port=0
--tserver_master_addrs=127.13.100.62:35021
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.2.keytab
--principal=kudu/127.13.100.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.100.20:34373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707036041078-13712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707036041078-13712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707036041078-13712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:41:28.316329 655 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:41:28.316613 655 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:41:28.316686 655 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:41:28.320793 655 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:41:28.320887 655 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:41:28.320981 655 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.100.2
I20260929 18:41:28.326970 655 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.100.20:34373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707036041078-13712-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707036041078-13712-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.2.keytab
--principal=kudu/127.13.100.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.100.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707036041078-13712-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.100.2
--webserver_port=0
--tserver_master_addrs=127.13.100.62:35021
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.655
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.100.2
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707036041078-13712-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:41:28.327953 655 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:41:28.328905 655 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:41:28.338068 661 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:41:28.338057 660 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:41:28.338174 663 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:41:28.338310 655 server_base.cc:1034] running on GCE node
I20260929 18:41:28.338873 655 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:41:28.339509 655 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:41:28.340698 655 hybrid_clock.cc:648] HybridClock initialized: now 1790707288340684 us; error 30 us; skew 500 ppm
Sep 29 18:41:28 dist-test-slave-j1xs krb5kdc[414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707288, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:41:28.343434 655 init.cc:377] Logged in from keytab as kudu/127.13.100.2@KRBTEST.COM (short username kudu)
I20260929 18:41:28.344359 655 webserver.cc:460] Webserver started at http://127.13.100.2:34615/ using document root <none> and password file <none>
I20260929 18:41:28.344901 655 fs_manager.cc:359] Metadata directory not provided
I20260929 18:41:28.344987 655 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:41:28.345211 655 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:41:28.347247 655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707036041078-13712-0/minicluster-data/ts-1/data/instance:
uuid: "cbe3caf16ef444fd91af87498a4d15b2"
format_stamp: "Formatted at 2026-09-29 18:41:28 on dist-test-slave-j1xs"
server_key: "c2b6a13fb02d516c25d1f452d9d1431b"
server_key_iv: "71729f8f96a8685fcae9885b259ee831"
server_key_version: "encryptionkey@0"
I20260929 18:41:28.347803 655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707036041078-13712-0/minicluster-data/ts-1/wal/instance:
uuid: "cbe3caf16ef444fd91af87498a4d15b2"
format_stamp: "Formatted at 2026-09-29 18:41:28 on dist-test-slave-j1xs"
server_key: "c2b6a13fb02d516c25d1f452d9d1431b"
server_key_iv: "71729f8f96a8685fcae9885b259ee831"
server_key_version: "encryptionkey@0"
I20260929 18:41:28.351469 655 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.001s
I20260929 18:41:28.353750 670 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:41:28.354940 655 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260929 18:41:28.355079 655 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707036041078-13712-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707036041078-13712-0/minicluster-data/ts-1/wal
uuid: "cbe3caf16ef444fd91af87498a4d15b2"
format_stamp: "Formatted at 2026-09-29 18:41:28 on dist-test-slave-j1xs"
server_key: "c2b6a13fb02d516c25d1f452d9d1431b"
server_key_iv: "71729f8f96a8685fcae9885b259ee831"
server_key_version: "encryptionkey@0"
I20260929 18:41:28.355188 655 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707036041078-13712-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707036041078-13712-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707036041078-13712-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:41:28.366185 655 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:41:28.369030 655 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:41:28.369235 655 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:41:28.369813 655 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:41:28.370641 655 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:41:28.370714 655 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:41:28.370780 655 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:41:28.370834 655 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:41:28.379841 655 rpc_server.cc:307] RPC server started. Bound to: 127.13.100.2:45397
I20260929 18:41:28.379889 783 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.100.2:45397 every 8 connection(s)
I20260929 18:41:28.380759 655 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707036041078-13712-0/minicluster-data/ts-1/data/info.pb
I20260929 18:41:28.384757 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 655
I20260929 18:41:28.384919 13712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707036041078-13712-0/minicluster-data/ts-1/wal/instance
I20260929 18:41:28.385205 13712 external_mini_cluster.cc:1428] Setting key e89c8b159a077b460ffbde78f3fb6931
Sep 29 18:41:28 dist-test-slave-j1xs krb5kdc[414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707288, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.2@KRBTEST.COM for kudu/127.13.100.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.13.100.3@KRBTEST.COM; defaulting to no policy
I20260929 18:41:28.396051 498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:28.382601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.2:34221 (local address 127.13.100.62:35021)
0929 18:41:28.382786 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0929 18:41:28.382790 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:41:28.383602 (+ 812us) server_negotiation.cc:407] Connection header received
0929 18:41:28.385185 (+ 1583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:41:28.385188 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:41:28.385246 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:41:28.385352 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:41:28.386649 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:28.387159 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:28.388752 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:28.389161 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:28.389304 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:28.392032 (+ 2728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:41:28.392039 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:41:28.392044 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:41:28.392079 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:41:28.393753 (+ 1674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:41:28.394313 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:41:28.394317 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:41:28.394319 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:41:28.394415 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:41:28.394757 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:41:28.394763 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:41:28.394767 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:41:28.395115 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:41:28.395236 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0929 18:41:28.395818 (+ 582us) server_negotiation.cc:299] Negotiation successful
0929 18:41:28.395924 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260929 18:41:28.396890 786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:28.382962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:35021 (local address 127.13.100.2:34221)
0929 18:41:28.383452 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:28.383505 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 18:41:28.384915 (+ 1410us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:41:28.385553 (+ 638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:41:28.385562 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:41:28.386023 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:41:28.386491 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:41:28.386504 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:41:28.387297 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:41:28.387304 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:41:28.388605 (+ 1301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:41:28.388615 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:41:28.389331 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:41:28.389333 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:41:28.389475 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:28.390172 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:41:28.390188 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:41:28.391856 (+ 1668us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:41:28.393873 (+ 2017us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:41:28.393877 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:41:28.393948 (+ 71us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:41:28.394204 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:41:28.394515 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:41:28.394517 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:41:28.394519 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:41:28.394642 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:41:28.395263 (+ 621us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:41:28.395268 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:41:28.395555 (+ 287us) client_negotiation.cc:769] Sending connection context
0929 18:41:28.395712 (+ 157us) client_negotiation.cc:240] Negotiation successful
0929 18:41:28.395970 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":103,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.100.3@KRBTEST.COM" created.
I20260929 18:41:28.398279 784 heartbeater.cc:347] Connected to a master server at 127.13.100.62:35021
I20260929 18:41:28.398530 784 heartbeater.cc:464] Registering TS with master...
I20260929 18:41:28.399050 784 heartbeater.cc:511] Master 127.13.100.62:35021 requested a full tablet report, sending...
I20260929 18:41:28.400164 459 ts_manager.cc:194] Registered new tserver with Master: cbe3caf16ef444fd91af87498a4d15b2 (127.13.100.2:45397)
I20260929 18:41:28.400751 459 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.100.2@KRBTEST.COM'} at 127.13.100.2:34221
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.100.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.3.keytab.
Entry for principal kudu/127.13.100.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.3.keytab.
I20260929 18:41:28.413407 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707036041078-13712-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707036041078-13712-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707036041078-13712-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707036041078-13712-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.100.3:0
--local_ip_for_outbound_sockets=127.13.100.3
--webserver_interface=127.13.100.3
--webserver_port=0
--tserver_master_addrs=127.13.100.62:35021
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.3.keytab
--principal=kudu/127.13.100.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.100.20:34373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707036041078-13712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707036041078-13712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707036041078-13712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:41:28.523434 789 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:41:28.523694 789 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:41:28.523746 789 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:41:28.527989 789 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:41:28.528044 789 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:41:28.528105 789 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.100.3
I20260929 18:41:28.533540 789 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.100.20:34373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707036041078-13712-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707036041078-13712-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.3.keytab
--principal=kudu/127.13.100.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.100.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707036041078-13712-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.100.3
--webserver_port=0
--tserver_master_addrs=127.13.100.62:35021
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.789
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.100.3
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707036041078-13712-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:41:28.534740 789 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:41:28.535677 789 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:41:28.544963 795 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:41:28.544984 797 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:41:28.544961 794 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:41:28.545291 789 server_base.cc:1034] running on GCE node
I20260929 18:41:28.545683 789 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:41:28.546401 789 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:41:28.547595 789 hybrid_clock.cc:648] HybridClock initialized: now 1790707288547587 us; error 31 us; skew 500 ppm
Sep 29 18:41:28 dist-test-slave-j1xs krb5kdc[414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707288, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:41:28.550485 789 init.cc:377] Logged in from keytab as kudu/127.13.100.3@KRBTEST.COM (short username kudu)
I20260929 18:41:28.551333 789 webserver.cc:460] Webserver started at http://127.13.100.3:44303/ using document root <none> and password file <none>
I20260929 18:41:28.551971 789 fs_manager.cc:359] Metadata directory not provided
I20260929 18:41:28.552059 789 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:41:28.552268 789 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:41:28.554363 789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707036041078-13712-0/minicluster-data/ts-2/data/instance:
uuid: "806ec05781fd4a64b2f210297cfa522b"
format_stamp: "Formatted at 2026-09-29 18:41:28 on dist-test-slave-j1xs"
server_key: "82645d4633418d5bf197a3a9c1ca5da5"
server_key_iv: "38cd1f061f6e6005e76c3c9d7e72dfbe"
server_key_version: "encryptionkey@0"
I20260929 18:41:28.554937 789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707036041078-13712-0/minicluster-data/ts-2/wal/instance:
uuid: "806ec05781fd4a64b2f210297cfa522b"
format_stamp: "Formatted at 2026-09-29 18:41:28 on dist-test-slave-j1xs"
server_key: "82645d4633418d5bf197a3a9c1ca5da5"
server_key_iv: "38cd1f061f6e6005e76c3c9d7e72dfbe"
server_key_version: "encryptionkey@0"
I20260929 18:41:28.558952 789 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 18:41:28.561573 804 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:41:28.563004 789 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:41:28.563153 789 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707036041078-13712-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707036041078-13712-0/minicluster-data/ts-2/wal
uuid: "806ec05781fd4a64b2f210297cfa522b"
format_stamp: "Formatted at 2026-09-29 18:41:28 on dist-test-slave-j1xs"
server_key: "82645d4633418d5bf197a3a9c1ca5da5"
server_key_iv: "38cd1f061f6e6005e76c3c9d7e72dfbe"
server_key_version: "encryptionkey@0"
I20260929 18:41:28.563285 789 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707036041078-13712-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707036041078-13712-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707036041078-13712-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:41:28.588536 789 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:41:28.591940 789 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:41:28.592150 789 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:41:28.592742 789 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:41:28.593567 789 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:41:28.593614 789 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:41:28.593681 789 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:41:28.593722 789 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:41:28.602772 789 rpc_server.cc:307] RPC server started. Bound to: 127.13.100.3:37009
I20260929 18:41:28.602795 917 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.100.3:37009 every 8 connection(s)
I20260929 18:41:28.603667 789 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707036041078-13712-0/minicluster-data/ts-2/data/info.pb
I20260929 18:41:28.610162 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 789
I20260929 18:41:28.610320 13712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707036041078-13712-0/minicluster-data/ts-2/wal/instance
I20260929 18:41:28.610622 13712 external_mini_cluster.cc:1428] Setting key a84e776c196ba771dbbd8983ebe0778f
Sep 29 18:41:28 dist-test-slave-j1xs krb5kdc[414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707288, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.3@KRBTEST.COM for kudu/127.13.100.62@KRBTEST.COM
I20260929 18:41:28.617061 498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:28.605415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.3:37531 (local address 127.13.100.62:35021)
0929 18:41:28.605560 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0929 18:41:28.605563 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:41:28.606478 (+ 915us) server_negotiation.cc:407] Connection header received
0929 18:41:28.607822 (+ 1344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:41:28.607825 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:41:28.607882 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:41:28.607958 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:41:28.609069 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:28.609606 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:28.610963 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:28.611189 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:28.611305 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:28.613781 (+ 2476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:41:28.613784 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:41:28.613786 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:41:28.613810 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:41:28.615089 (+ 1279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:41:28.615581 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:41:28.615587 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:41:28.615591 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:41:28.615670 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:41:28.616004 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:41:28.616009 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:41:28.616010 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:41:28.616184 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:41:28.616293 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 18:41:28.616766 (+ 473us) server_negotiation.cc:299] Negotiation successful
0929 18:41:28.616895 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 18:41:28.617865 920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:28.605822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:35021 (local address 127.13.100.3:37531)
0929 18:41:28.606310 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:28.606350 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 18:41:28.607608 (+ 1258us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:41:28.608115 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:41:28.608121 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:41:28.608463 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:41:28.608929 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:41:28.608941 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:41:28.609731 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:41:28.609734 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:41:28.610844 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:41:28.610854 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:41:28.611346 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:41:28.611350 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:41:28.611512 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:28.612127 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:41:28.612143 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:41:28.613611 (+ 1468us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:41:28.615204 (+ 1593us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:41:28.615210 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:41:28.615223 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:41:28.615476 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:41:28.615763 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:41:28.615768 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:41:28.615770 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:41:28.615900 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:41:28.616298 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:41:28.616306 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:41:28.616541 (+ 235us) client_negotiation.cc:769] Sending connection context
0929 18:41:28.616724 (+ 183us) client_negotiation.cc:240] Negotiation successful
0929 18:41:28.617027 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":106,"threads_started":1}
I20260929 18:41:28.619148 918 heartbeater.cc:347] Connected to a master server at 127.13.100.62:35021
I20260929 18:41:28.619388 918 heartbeater.cc:464] Registering TS with master...
I20260929 18:41:28.619855 918 heartbeater.cc:511] Master 127.13.100.62:35021 requested a full tablet report, sending...
I20260929 18:41:28.620870 459 ts_manager.cc:194] Registered new tserver with Master: 806ec05781fd4a64b2f210297cfa522b (127.13.100.3:37009)
I20260929 18:41:28.621444 459 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.100.3@KRBTEST.COM'} at 127.13.100.3:37531
I20260929 18:41:28.624166 13712 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 18:41:28.633438 498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:28.626124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41426 (local address 127.13.100.62:35021)
0929 18:41:28.626257 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0929 18:41:28.626262 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:41:28.626408 (+ 146us) server_negotiation.cc:407] Connection header received
0929 18:41:28.626588 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:41:28.626590 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:41:28.626626 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:41:28.626740 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:41:28.627437 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:28.627997 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:28.628785 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:28.628986 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:28.629061 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:28.630420 (+ 1359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:41:28.630423 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:41:28.630425 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:41:28.630446 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:41:28.631721 (+ 1275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:41:28.632142 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:41:28.632145 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:41:28.632146 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:41:28.632241 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:41:28.632538 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:41:28.632540 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:41:28.632541 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:41:28.632753 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:41:28.632862 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 18:41:28.633111 (+ 249us) server_negotiation.cc:299] Negotiation successful
0929 18:41:28.633257 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 18:41:28.636261 459 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41426:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:41:28.639197 459 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:41:28.653115 929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:28.647093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.3:37009 (local address 127.0.0.1:43014)
0929 18:41:28.647577 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:28.647620 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 18:41:28.647757 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:41:28.648546 (+ 789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:41:28.648551 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:41:28.648571 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:41:28.648705 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:41:28.648713 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:41:28.650141 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:41:28.650144 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:41:28.651450 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:41:28.651462 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:41:28.652573 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:41:28.652576 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:41:28.652705 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:28.652748 (+ 43us) client_negotiation.cc:769] Sending connection context
0929 18:41:28.652852 (+ 104us) client_negotiation.cc:240] Negotiation successful
0929 18:41:28.652963 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":89,"threads_started":1}
I20260929 18:41:28.653282 933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:28.648086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38166 (local address 127.13.100.2:45397)
0929 18:41:28.648962 (+ 876us) server_negotiation.cc:206] Beginning negotiation
0929 18:41:28.648973 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 18:41:28.648993 (+ 20us) server_negotiation.cc:407] Connection header received
0929 18:41:28.649060 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:41:28.649064 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:41:28.649224 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:41:28.649400 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:41:28.649591 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:28.650659 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:28.652007 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:28.652561 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:28.652853 (+ 292us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:28.652989 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0929 18:41:28.653038 (+ 49us) server_negotiation.cc:299] Negotiation successful
0929 18:41:28.653117 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":776,"thread_start_us":119,"threads_started":1}
I20260929 18:41:28.653169 932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:28.647775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.2:45397 (local address 127.0.0.1:38166)
0929 18:41:28.648443 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:28.648454 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 18:41:28.648559 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:41:28.649380 (+ 821us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:41:28.649383 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:41:28.649400 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:41:28.649475 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:41:28.649480 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:41:28.650845 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:41:28.650848 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:41:28.651898 (+ 1050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:41:28.651905 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:41:28.652686 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:41:28.652690 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:41:28.652770 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:28.652872 (+ 102us) client_negotiation.cc:769] Sending connection context
0929 18:41:28.652938 (+ 66us) client_negotiation.cc:240] Negotiation successful
0929 18:41:28.653019 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":609,"mutex_wait_us":52,"thread_start_us":86,"threads_started":1}
I20260929 18:41:28.653726 931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:28.647287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43014 (local address 127.13.100.3:37009)
0929 18:41:28.647881 (+ 594us) server_negotiation.cc:206] Beginning negotiation
0929 18:41:28.647897 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 18:41:28.647921 (+ 24us) server_negotiation.cc:407] Connection header received
0929 18:41:28.647998 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:41:28.648002 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:41:28.648203 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:41:28.648371 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:41:28.648849 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:28.650013 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:28.651623 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:28.652439 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:28.652558 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:28.652692 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0929 18:41:28.653510 (+ 818us) server_negotiation.cc:299] Negotiation successful
0929 18:41:28.653614 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":489,"thread_start_us":149,"threads_started":1}
I20260929 18:41:28.653753 934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:28.648134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41522 (local address 127.13.100.1:42395)
0929 18:41:28.648657 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0929 18:41:28.648673 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 18:41:28.648694 (+ 21us) server_negotiation.cc:407] Connection header received
0929 18:41:28.648763 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:41:28.648770 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:41:28.648941 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:41:28.649083 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:41:28.649312 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:28.650285 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:28.651762 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:28.652485 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:28.652570 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:28.652692 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0929 18:41:28.653178 (+ 486us) server_negotiation.cc:299] Negotiation successful
0929 18:41:28.653597 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":431,"thread_start_us":85,"threads_started":1}
I20260929 18:41:28.653983 930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:28.647283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.1:42395 (local address 127.0.0.1:41522)
0929 18:41:28.647782 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:28.647796 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:41:28.647887 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:41:28.649075 (+ 1188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:41:28.649077 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:41:28.649094 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:41:28.649188 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:41:28.649196 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:41:28.650420 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:41:28.650423 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:41:28.651637 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:41:28.651646 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:41:28.652773 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:41:28.652776 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:41:28.652949 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:28.652968 (+ 19us) client_negotiation.cc:769] Sending connection context
0929 18:41:28.653026 (+ 58us) client_negotiation.cc:240] Negotiation successful
0929 18:41:28.653077 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":421,"mutex_wait_us":20,"thread_start_us":98,"threads_started":1}
I20260929 18:41:28.656088 718 tablet_service.cc:1467] Processing CreateTablet for tablet c2d03614b714484da9ff4b486fe30c10 (DEFAULT_TABLE table=test-table [id=e6514b96def248248469ae95b8d274f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:41:28.656423 852 tablet_service.cc:1467] Processing CreateTablet for tablet c2d03614b714484da9ff4b486fe30c10 (DEFAULT_TABLE table=test-table [id=e6514b96def248248469ae95b8d274f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:41:28.656426 584 tablet_service.cc:1467] Processing CreateTablet for tablet c2d03614b714484da9ff4b486fe30c10 (DEFAULT_TABLE table=test-table [id=e6514b96def248248469ae95b8d274f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:41:28.657068 718 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2d03614b714484da9ff4b486fe30c10. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:41:28.657359 584 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2d03614b714484da9ff4b486fe30c10. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:41:28.657363 852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2d03614b714484da9ff4b486fe30c10. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:41:28.663678 935 tablet_bootstrap.cc:492] T c2d03614b714484da9ff4b486fe30c10 P cbe3caf16ef444fd91af87498a4d15b2: Bootstrap starting.
I20260929 18:41:28.664759 936 tablet_bootstrap.cc:492] T c2d03614b714484da9ff4b486fe30c10 P 791a030d2d3f4d5ab760fd83cf19cdf5: Bootstrap starting.
I20260929 18:41:28.665732 935 tablet_bootstrap.cc:654] T c2d03614b714484da9ff4b486fe30c10 P cbe3caf16ef444fd91af87498a4d15b2: Neither blocks nor log segments found. Creating new log.
I20260929 18:41:28.666288 937 tablet_bootstrap.cc:492] T c2d03614b714484da9ff4b486fe30c10 P 806ec05781fd4a64b2f210297cfa522b: Bootstrap starting.
I20260929 18:41:28.666563 936 tablet_bootstrap.cc:654] T c2d03614b714484da9ff4b486fe30c10 P 791a030d2d3f4d5ab760fd83cf19cdf5: Neither blocks nor log segments found. Creating new log.
I20260929 18:41:28.667188 935 log.cc:824] T c2d03614b714484da9ff4b486fe30c10 P cbe3caf16ef444fd91af87498a4d15b2: Log is configured to *not* fsync() on all Append() calls
I20260929 18:41:28.667758 936 log.cc:824] T c2d03614b714484da9ff4b486fe30c10 P 791a030d2d3f4d5ab760fd83cf19cdf5: Log is configured to *not* fsync() on all Append() calls
I20260929 18:41:28.668358 937 tablet_bootstrap.cc:654] T c2d03614b714484da9ff4b486fe30c10 P 806ec05781fd4a64b2f210297cfa522b: Neither blocks nor log segments found. Creating new log.
I20260929 18:41:28.669293 935 tablet_bootstrap.cc:492] T c2d03614b714484da9ff4b486fe30c10 P cbe3caf16ef444fd91af87498a4d15b2: No bootstrap required, opened a new log
I20260929 18:41:28.669432 935 ts_tablet_manager.cc:1397] T c2d03614b714484da9ff4b486fe30c10 P cbe3caf16ef444fd91af87498a4d15b2: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260929 18:41:28.669549 937 log.cc:824] T c2d03614b714484da9ff4b486fe30c10 P 806ec05781fd4a64b2f210297cfa522b: Log is configured to *not* fsync() on all Append() calls
I20260929 18:41:28.669927 936 tablet_bootstrap.cc:492] T c2d03614b714484da9ff4b486fe30c10 P 791a030d2d3f4d5ab760fd83cf19cdf5: No bootstrap required, opened a new log
I20260929 18:41:28.670078 936 ts_tablet_manager.cc:1397] T c2d03614b714484da9ff4b486fe30c10 P 791a030d2d3f4d5ab760fd83cf19cdf5: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260929 18:41:28.671322 937 tablet_bootstrap.cc:492] T c2d03614b714484da9ff4b486fe30c10 P 806ec05781fd4a64b2f210297cfa522b: No bootstrap required, opened a new log
I20260929 18:41:28.671476 937 ts_tablet_manager.cc:1397] T c2d03614b714484da9ff4b486fe30c10 P 806ec05781fd4a64b2f210297cfa522b: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 18:41:28.672742 936 raft_consensus.cc:348] T c2d03614b714484da9ff4b486fe30c10 P 791a030d2d3f4d5ab760fd83cf19cdf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "806ec05781fd4a64b2f210297cfa522b" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 37009 } } peers { permanent_uuid: "791a030d2d3f4d5ab760fd83cf19cdf5" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 42395 } } peers { permanent_uuid: "cbe3caf16ef444fd91af87498a4d15b2" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 45397 } }
I20260929 18:41:28.672938 936 raft_consensus.cc:374] T c2d03614b714484da9ff4b486fe30c10 P 791a030d2d3f4d5ab760fd83cf19cdf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:41:28.672968 936 raft_consensus.cc:729] T c2d03614b714484da9ff4b486fe30c10 P 791a030d2d3f4d5ab760fd83cf19cdf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 791a030d2d3f4d5ab760fd83cf19cdf5, State: Initialized, Role: FOLLOWER
I20260929 18:41:28.673172 935 raft_consensus.cc:348] T c2d03614b714484da9ff4b486fe30c10 P cbe3caf16ef444fd91af87498a4d15b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "806ec05781fd4a64b2f210297cfa522b" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 37009 } } peers { permanent_uuid: "791a030d2d3f4d5ab760fd83cf19cdf5" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 42395 } } peers { permanent_uuid: "cbe3caf16ef444fd91af87498a4d15b2" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 45397 } }
I20260929 18:41:28.673427 935 raft_consensus.cc:374] T c2d03614b714484da9ff4b486fe30c10 P cbe3caf16ef444fd91af87498a4d15b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:41:28.673389 936 consensus_queue.cc:260] T c2d03614b714484da9ff4b486fe30c10 P 791a030d2d3f4d5ab760fd83cf19cdf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "806ec05781fd4a64b2f210297cfa522b" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 37009 } } peers { permanent_uuid: "791a030d2d3f4d5ab760fd83cf19cdf5" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 42395 } } peers { permanent_uuid: "cbe3caf16ef444fd91af87498a4d15b2" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 45397 } }
I20260929 18:41:28.673483 935 raft_consensus.cc:729] T c2d03614b714484da9ff4b486fe30c10 P cbe3caf16ef444fd91af87498a4d15b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbe3caf16ef444fd91af87498a4d15b2, State: Initialized, Role: FOLLOWER
I20260929 18:41:28.673920 935 consensus_queue.cc:260] T c2d03614b714484da9ff4b486fe30c10 P cbe3caf16ef444fd91af87498a4d15b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "806ec05781fd4a64b2f210297cfa522b" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 37009 } } peers { permanent_uuid: "791a030d2d3f4d5ab760fd83cf19cdf5" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 42395 } } peers { permanent_uuid: "cbe3caf16ef444fd91af87498a4d15b2" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 45397 } }
I20260929 18:41:28.674345 650 heartbeater.cc:503] Master 127.13.100.62:35021 was elected leader, sending a full tablet report...
I20260929 18:41:28.674422 936 ts_tablet_manager.cc:1428] T c2d03614b714484da9ff4b486fe30c10 P 791a030d2d3f4d5ab760fd83cf19cdf5: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260929 18:41:28.674795 935 ts_tablet_manager.cc:1428] T c2d03614b714484da9ff4b486fe30c10 P cbe3caf16ef444fd91af87498a4d15b2: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260929 18:41:28.675115 784 heartbeater.cc:503] Master 127.13.100.62:35021 was elected leader, sending a full tablet report...
I20260929 18:41:28.675290 937 raft_consensus.cc:348] T c2d03614b714484da9ff4b486fe30c10 P 806ec05781fd4a64b2f210297cfa522b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "806ec05781fd4a64b2f210297cfa522b" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 37009 } } peers { permanent_uuid: "791a030d2d3f4d5ab760fd83cf19cdf5" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 42395 } } peers { permanent_uuid: "cbe3caf16ef444fd91af87498a4d15b2" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 45397 } }
I20260929 18:41:28.675529 937 raft_consensus.cc:374] T c2d03614b714484da9ff4b486fe30c10 P 806ec05781fd4a64b2f210297cfa522b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:41:28.675590 937 raft_consensus.cc:729] T c2d03614b714484da9ff4b486fe30c10 P 806ec05781fd4a64b2f210297cfa522b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 806ec05781fd4a64b2f210297cfa522b, State: Initialized, Role: FOLLOWER
I20260929 18:41:28.675995 937 consensus_queue.cc:260] T c2d03614b714484da9ff4b486fe30c10 P 806ec05781fd4a64b2f210297cfa522b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "806ec05781fd4a64b2f210297cfa522b" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 37009 } } peers { permanent_uuid: "791a030d2d3f4d5ab760fd83cf19cdf5" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 42395 } } peers { permanent_uuid: "cbe3caf16ef444fd91af87498a4d15b2" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 45397 } }
I20260929 18:41:28.676815 918 heartbeater.cc:503] Master 127.13.100.62:35021 was elected leader, sending a full tablet report...
I20260929 18:41:28.676921 937 ts_tablet_manager.cc:1428] T c2d03614b714484da9ff4b486fe30c10 P 806ec05781fd4a64b2f210297cfa522b: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20260929 18:41:28.677683 651 tablet.cc:2369] T c2d03614b714484da9ff4b486fe30c10 P 791a030d2d3f4d5ab760fd83cf19cdf5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:41:28.701750 941 raft_consensus.cc:482] T c2d03614b714484da9ff4b486fe30c10 P 791a030d2d3f4d5ab760fd83cf19cdf5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:41:28.702064 941 raft_consensus.cc:504] T c2d03614b714484da9ff4b486fe30c10 P 791a030d2d3f4d5ab760fd83cf19cdf5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "806ec05781fd4a64b2f210297cfa522b" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 37009 } } peers { permanent_uuid: "791a030d2d3f4d5ab760fd83cf19cdf5" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 42395 } } peers { permanent_uuid: "cbe3caf16ef444fd91af87498a4d15b2" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 45397 } }
I20260929 18:41:28.703292 941 leader_election.cc:288] T c2d03614b714484da9ff4b486fe30c10 P 791a030d2d3f4d5ab760fd83cf19cdf5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 806ec05781fd4a64b2f210297cfa522b (127.13.100.3:37009), cbe3caf16ef444fd91af87498a4d15b2 (127.13.100.2:45397)
I20260929 18:41:28.707464 944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:28.703580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.3:37009 (local address 127.13.100.1:42599)
0929 18:41:28.703892 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:28.703910 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 18:41:28.704070 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:41:28.704396 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:41:28.704399 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:41:28.704418 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:41:28.704562 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:41:28.704568 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:41:28.705337 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:41:28.705344 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:41:28.706283 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:41:28.706293 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:41:28.706974 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:41:28.706979 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:41:28.707087 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:28.707137 (+ 50us) client_negotiation.cc:769] Sending connection context
0929 18:41:28.707231 (+ 94us) client_negotiation.cc:240] Negotiation successful
0929 18:41:28.707313 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"mutex_wait_us":1,"thread_start_us":107,"threads_started":1}
I20260929 18:41:28.707480 933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:28.704129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.1:60231 (local address 127.13.100.2:45397)
0929 18:41:28.704271 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0929 18:41:28.704276 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:41:28.704443 (+ 167us) server_negotiation.cc:407] Connection header received
0929 18:41:28.704590 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:41:28.704592 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:41:28.704633 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:41:28.704719 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:41:28.705010 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:28.705502 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:28.706532 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:28.706930 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:28.707054 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:28.707104 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0929 18:41:28.707324 (+ 220us) server_negotiation.cc:299] Negotiation successful
0929 18:41:28.707384 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 18:41:28.707469 931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:28.703724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.1:42599 (local address 127.13.100.3:37009)
0929 18:41:28.703886 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0929 18:41:28.703891 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:41:28.704013 (+ 122us) server_negotiation.cc:407] Connection header received
0929 18:41:28.704234 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:41:28.704238 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:41:28.704290 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:41:28.704381 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:41:28.704722 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:28.705218 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:28.706421 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:28.706845 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:28.706986 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:28.707046 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0929 18:41:28.707267 (+ 221us) server_negotiation.cc:299] Negotiation successful
0929 18:41:28.707348 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 18:41:28.708029 945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:28.703998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.2:45397 (local address 127.13.100.1:60231)
0929 18:41:28.704354 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:28.704364 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 18:41:28.704481 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:41:28.704729 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:41:28.704731 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:41:28.704786 (+ 55us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:41:28.704899 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:41:28.704904 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:41:28.705619 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:41:28.705624 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:41:28.706412 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:41:28.706420 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:41:28.707049 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:41:28.707052 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:41:28.707161 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:28.707182 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 18:41:28.707791 (+ 609us) client_negotiation.cc:240] Negotiation successful
0929 18:41:28.707865 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":99,"threads_started":1}
I20260929 18:41:28.708297 872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2d03614b714484da9ff4b486fe30c10" candidate_uuid: "791a030d2d3f4d5ab760fd83cf19cdf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "806ec05781fd4a64b2f210297cfa522b" is_pre_election: true
I20260929 18:41:28.708631 738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2d03614b714484da9ff4b486fe30c10" candidate_uuid: "791a030d2d3f4d5ab760fd83cf19cdf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cbe3caf16ef444fd91af87498a4d15b2" is_pre_election: true
I20260929 18:41:28.708678 872 raft_consensus.cc:2454] T c2d03614b714484da9ff4b486fe30c10 P 806ec05781fd4a64b2f210297cfa522b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 791a030d2d3f4d5ab760fd83cf19cdf5 in term 0.
I20260929 18:41:28.708904 738 raft_consensus.cc:2454] T c2d03614b714484da9ff4b486fe30c10 P cbe3caf16ef444fd91af87498a4d15b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 791a030d2d3f4d5ab760fd83cf19cdf5 in term 0.
I20260929 18:41:28.709237 540 leader_election.cc:302] T c2d03614b714484da9ff4b486fe30c10 P 791a030d2d3f4d5ab760fd83cf19cdf5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 791a030d2d3f4d5ab760fd83cf19cdf5, 806ec05781fd4a64b2f210297cfa522b; no voters:
I20260929 18:41:28.709529 941 raft_consensus.cc:2781] T c2d03614b714484da9ff4b486fe30c10 P 791a030d2d3f4d5ab760fd83cf19cdf5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:41:28.709618 941 raft_consensus.cc:482] T c2d03614b714484da9ff4b486fe30c10 P 791a030d2d3f4d5ab760fd83cf19cdf5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:41:28.709661 941 raft_consensus.cc:3037] T c2d03614b714484da9ff4b486fe30c10 P 791a030d2d3f4d5ab760fd83cf19cdf5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:41:28.710740 941 raft_consensus.cc:504] T c2d03614b714484da9ff4b486fe30c10 P 791a030d2d3f4d5ab760fd83cf19cdf5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "806ec05781fd4a64b2f210297cfa522b" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 37009 } } peers { permanent_uuid: "791a030d2d3f4d5ab760fd83cf19cdf5" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 42395 } } peers { permanent_uuid: "cbe3caf16ef444fd91af87498a4d15b2" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 45397 } }
I20260929 18:41:28.711072 941 leader_election.cc:288] T c2d03614b714484da9ff4b486fe30c10 P 791a030d2d3f4d5ab760fd83cf19cdf5 [CANDIDATE]: Term 1 election: Requested vote from peers 806ec05781fd4a64b2f210297cfa522b (127.13.100.3:37009), cbe3caf16ef444fd91af87498a4d15b2 (127.13.100.2:45397)
I20260929 18:41:28.711483 872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2d03614b714484da9ff4b486fe30c10" candidate_uuid: "791a030d2d3f4d5ab760fd83cf19cdf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "806ec05781fd4a64b2f210297cfa522b"
I20260929 18:41:28.711483 738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2d03614b714484da9ff4b486fe30c10" candidate_uuid: "791a030d2d3f4d5ab760fd83cf19cdf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cbe3caf16ef444fd91af87498a4d15b2"
I20260929 18:41:28.711613 872 raft_consensus.cc:3037] T c2d03614b714484da9ff4b486fe30c10 P 806ec05781fd4a64b2f210297cfa522b [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:41:28.711614 738 raft_consensus.cc:3037] T c2d03614b714484da9ff4b486fe30c10 P cbe3caf16ef444fd91af87498a4d15b2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:41:28.712913 872 raft_consensus.cc:2454] T c2d03614b714484da9ff4b486fe30c10 P 806ec05781fd4a64b2f210297cfa522b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 791a030d2d3f4d5ab760fd83cf19cdf5 in term 1.
I20260929 18:41:28.712913 738 raft_consensus.cc:2454] T c2d03614b714484da9ff4b486fe30c10 P cbe3caf16ef444fd91af87498a4d15b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 791a030d2d3f4d5ab760fd83cf19cdf5 in term 1.
I20260929 18:41:28.713274 540 leader_election.cc:302] T c2d03614b714484da9ff4b486fe30c10 P 791a030d2d3f4d5ab760fd83cf19cdf5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 791a030d2d3f4d5ab760fd83cf19cdf5, 806ec05781fd4a64b2f210297cfa522b; no voters:
I20260929 18:41:28.713435 941 raft_consensus.cc:2781] T c2d03614b714484da9ff4b486fe30c10 P 791a030d2d3f4d5ab760fd83cf19cdf5 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:41:28.713670 941 raft_consensus.cc:686] T c2d03614b714484da9ff4b486fe30c10 P 791a030d2d3f4d5ab760fd83cf19cdf5 [term 1 LEADER]: Becoming Leader. State: Replica: 791a030d2d3f4d5ab760fd83cf19cdf5, State: Running, Role: LEADER
I20260929 18:41:28.714083 941 consensus_queue.cc:237] T c2d03614b714484da9ff4b486fe30c10 P 791a030d2d3f4d5ab760fd83cf19cdf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "806ec05781fd4a64b2f210297cfa522b" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 37009 } } peers { permanent_uuid: "791a030d2d3f4d5ab760fd83cf19cdf5" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 42395 } } peers { permanent_uuid: "cbe3caf16ef444fd91af87498a4d15b2" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 45397 } }
I20260929 18:41:28.717330 457 catalog_manager.cc:5495] T c2d03614b714484da9ff4b486fe30c10 P 791a030d2d3f4d5ab760fd83cf19cdf5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 791a030d2d3f4d5ab760fd83cf19cdf5 (127.13.100.1). New cstate: current_term: 1 leader_uuid: "791a030d2d3f4d5ab760fd83cf19cdf5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "806ec05781fd4a64b2f210297cfa522b" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 37009 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "791a030d2d3f4d5ab760fd83cf19cdf5" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 42395 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cbe3caf16ef444fd91af87498a4d15b2" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 45397 } health_report { overall_health: UNKNOWN } } }
I20260929 18:41:28.731931 934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:28.728459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41538 (local address 127.13.100.1:42395)
0929 18:41:28.728590 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0929 18:41:28.728594 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:41:28.728607 (+ 13us) server_negotiation.cc:407] Connection header received
0929 18:41:28.728735 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:41:28.728737 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:41:28.728787 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:41:28.728886 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:41:28.729162 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:28.729650 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:28.730625 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:28.730866 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:28.730947 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:28.731237 (+ 290us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:41:28.731615 (+ 378us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:41:28.731728 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 18:41:28.731784 (+ 56us) server_negotiation.cc:299] Negotiation successful
0929 18:41:28.731829 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 18:41:28.737574 738 raft_consensus.cc:1261] T c2d03614b714484da9ff4b486fe30c10 P cbe3caf16ef444fd91af87498a4d15b2 [term 1 FOLLOWER]: Refusing update from remote peer 791a030d2d3f4d5ab760fd83cf19cdf5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:41:28.737608 872 raft_consensus.cc:1261] T c2d03614b714484da9ff4b486fe30c10 P 806ec05781fd4a64b2f210297cfa522b [term 1 FOLLOWER]: Refusing update from remote peer 791a030d2d3f4d5ab760fd83cf19cdf5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:41:28.738446 941 consensus_queue.cc:1035] T c2d03614b714484da9ff4b486fe30c10 P 791a030d2d3f4d5ab760fd83cf19cdf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbe3caf16ef444fd91af87498a4d15b2" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 45397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:41:28.738653 946 consensus_queue.cc:1035] T c2d03614b714484da9ff4b486fe30c10 P 791a030d2d3f4d5ab760fd83cf19cdf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "806ec05781fd4a64b2f210297cfa522b" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 37009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:41:28.744894 949 mvcc.cc:205] Tried to move back new op lower bound from 7334737054662295552 to 7334737054575538176. Current Snapshot: MvccSnapshot[applied={T|T < 7334737054662295552}]
I20260929 18:41:28.745313 950 mvcc.cc:205] Tried to move back new op lower bound from 7334737054662295552 to 7334737054575538176. Current Snapshot: MvccSnapshot[applied={T|T < 7334737054662295552}]
I20260929 18:41:28.746512 951 mvcc.cc:205] Tried to move back new op lower bound from 7334737054662295552 to 7334737054575538176. Current Snapshot: MvccSnapshot[applied={T|T < 7334737054662295552}]
I20260929 18:41:28.754832 458 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41426:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:41:28.755038 458 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41426:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:41:28.757719 458 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 320d980920d94ca8bb71b32ae1729865: Sending DeleteTablet for 3 replicas of tablet c2d03614b714484da9ff4b486fe30c10
I20260929 18:41:28.758654 584 tablet_service.cc:1514] Processing DeleteTablet for tablet c2d03614b714484da9ff4b486fe30c10 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:41:28 UTC) from {username='kudu', principal='kudu/127.13.100.62@KRBTEST.COM'} at 127.0.0.1:41522
I20260929 18:41:28.758646 852 tablet_service.cc:1514] Processing DeleteTablet for tablet c2d03614b714484da9ff4b486fe30c10 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:41:28 UTC) from {username='kudu', principal='kudu/127.13.100.62@KRBTEST.COM'} at 127.0.0.1:43014
I20260929 18:41:28.759248 959 tablet_replica.cc:330] T c2d03614b714484da9ff4b486fe30c10 P 791a030d2d3f4d5ab760fd83cf19cdf5: stopping tablet replica
I20260929 18:41:28.759222 718 tablet_service.cc:1514] Processing DeleteTablet for tablet c2d03614b714484da9ff4b486fe30c10 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:41:28 UTC) from {username='kudu', principal='kudu/127.13.100.62@KRBTEST.COM'} at 127.0.0.1:38166
I20260929 18:41:28.759591 959 raft_consensus.cc:2229] T c2d03614b714484da9ff4b486fe30c10 P 791a030d2d3f4d5ab760fd83cf19cdf5 [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:41:28.759797 960 tablet_replica.cc:330] T c2d03614b714484da9ff4b486fe30c10 P cbe3caf16ef444fd91af87498a4d15b2: stopping tablet replica
I20260929 18:41:28.760020 959 raft_consensus.cc:2258] T c2d03614b714484da9ff4b486fe30c10 P 791a030d2d3f4d5ab760fd83cf19cdf5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:41:28.760043 958 tablet_replica.cc:330] T c2d03614b714484da9ff4b486fe30c10 P 806ec05781fd4a64b2f210297cfa522b: stopping tablet replica
I20260929 18:41:28.760350 958 raft_consensus.cc:2229] T c2d03614b714484da9ff4b486fe30c10 P 806ec05781fd4a64b2f210297cfa522b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:41:28.760581 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 521
I20260929 18:41:28.760675 958 raft_consensus.cc:2258] T c2d03614b714484da9ff4b486fe30c10 P 806ec05781fd4a64b2f210297cfa522b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:41:28.762156 960 raft_consensus.cc:2229] T c2d03614b714484da9ff4b486fe30c10 P cbe3caf16ef444fd91af87498a4d15b2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:41:28.762458 960 raft_consensus.cc:2258] T c2d03614b714484da9ff4b486fe30c10 P cbe3caf16ef444fd91af87498a4d15b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:41:28.763768 960 ts_tablet_manager.cc:1905] T c2d03614b714484da9ff4b486fe30c10 P cbe3caf16ef444fd91af87498a4d15b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:41:28.763768 958 ts_tablet_manager.cc:1905] T c2d03614b714484da9ff4b486fe30c10 P 806ec05781fd4a64b2f210297cfa522b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:41:28.766484 958 ts_tablet_manager.cc:1918] T c2d03614b714484da9ff4b486fe30c10 P 806ec05781fd4a64b2f210297cfa522b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:41:28.766484 960 ts_tablet_manager.cc:1918] T c2d03614b714484da9ff4b486fe30c10 P cbe3caf16ef444fd91af87498a4d15b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:41:28.766577 958 log.cc:1196] T c2d03614b714484da9ff4b486fe30c10 P 806ec05781fd4a64b2f210297cfa522b: Deleting WAL directory at /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707036041078-13712-0/minicluster-data/ts-2/wal/wals/c2d03614b714484da9ff4b486fe30c10
I20260929 18:41:28.766621 960 log.cc:1196] T c2d03614b714484da9ff4b486fe30c10 P cbe3caf16ef444fd91af87498a4d15b2: Deleting WAL directory at /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707036041078-13712-0/minicluster-data/ts-1/wal/wals/c2d03614b714484da9ff4b486fe30c10
I20260929 18:41:28.766942 958 ts_tablet_manager.cc:1939] T c2d03614b714484da9ff4b486fe30c10 P 806ec05781fd4a64b2f210297cfa522b: Deleting consensus metadata
I20260929 18:41:28.767014 960 ts_tablet_manager.cc:1939] T c2d03614b714484da9ff4b486fe30c10 P cbe3caf16ef444fd91af87498a4d15b2: Deleting consensus metadata
W20260929 18:41:28.766981 443 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.100.1:42395 (error 108)
I20260929 18:41:28.768045 930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:28.767715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.1:42395 (local address 127.0.0.1:41544)
0929 18:41:28.767847 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:28.767922 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.1:42395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260929 18:41:28.768071 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 655
I20260929 18:41:28.768132 446 catalog_manager.cc:4841] TS 806ec05781fd4a64b2f210297cfa522b (127.13.100.3:37009): tablet c2d03614b714484da9ff4b486fe30c10 (table test-table [id=e6514b96def248248469ae95b8d274f8]) successfully deleted
W20260929 18:41:28.769675 443 catalog_manager.cc:4584] TS 791a030d2d3f4d5ab760fd83cf19cdf5 (127.13.100.1:42395): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c2d03614b714484da9ff4b486fe30c10: Network error: Client connection negotiation failed: client connection to 127.13.100.1:42395: connect: Connection refused (error 111)
I20260929 18:41:28.773581 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 789
I20260929 18:41:28.775169 930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:28.773727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.2:45397 (local address 127.0.0.1:38170)
0929 18:41:28.775015 (+ 1288us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:28.775083 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.2:45397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1189}
I20260929 18:41:28.778865 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 427
2026-09-29T18:41:28Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2202 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-29T18:41:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:41:28Z Disabled control of system clock
I20260929 18:41:28.809023 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707036041078-13712-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707036041078-13712-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707036041078-13712-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.100.62:37343
--webserver_interface=127.13.100.62
--webserver_port=0
--builtin_ntp_servers=127.13.100.20:39531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.100.62:37343
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 18:41:28.918416 965 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:41:28.918682 965 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:41:28.918756 965 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:41:28.922674 965 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:41:28.922721 965 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:41:28.922737 965 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:41:28.922785 965 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:41:28.922816 965 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:41:28.927451 965 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.100.20:39531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707036041078-13712-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707036041078-13712-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.100.62:37343
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.100.62:37343
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.100.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.965
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707036041078-13712-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:41:28.928424 965 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:41:28.929721 965 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:41:28.937580 970 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:41:28.937619 971 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:41:28.937614 973 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:41:28.937992 965 server_base.cc:1034] running on GCE node
I20260929 18:41:28.938493 965 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:41:28.939572 965 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:41:28.940730 965 hybrid_clock.cc:648] HybridClock initialized: now 1790707288940713 us; error 29 us; skew 500 ppm
I20260929 18:41:28.942512 965 webserver.cc:460] Webserver started at http://127.13.100.62:35967/ using document root <none> and password file <none>
I20260929 18:41:28.943051 965 fs_manager.cc:359] Metadata directory not provided
I20260929 18:41:28.943109 965 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:41:28.943310 965 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:41:28.945225 965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707036041078-13712-0/minicluster-data/master-0/data/instance:
uuid: "874a58551e194bbdb820df39e57aac10"
format_stamp: "Formatted at 2026-09-29 18:41:28 on dist-test-slave-j1xs"
server_key: "fc16c8901cbec5ff07a71fa8cf17eca1"
server_key_iv: "b9e1bcfb4321aac62b776693eb8b92b4"
server_key_version: "encryptionkey@0"
I20260929 18:41:28.945779 965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707036041078-13712-0/minicluster-data/master-0/wal/instance:
uuid: "874a58551e194bbdb820df39e57aac10"
format_stamp: "Formatted at 2026-09-29 18:41:28 on dist-test-slave-j1xs"
server_key: "fc16c8901cbec5ff07a71fa8cf17eca1"
server_key_iv: "b9e1bcfb4321aac62b776693eb8b92b4"
server_key_version: "encryptionkey@0"
I20260929 18:41:28.949195 965 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 18:41:28.951519 979 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:41:28.952693 965 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 18:41:28.952833 965 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707036041078-13712-0/minicluster-data/master-0/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707036041078-13712-0/minicluster-data/master-0/wal
uuid: "874a58551e194bbdb820df39e57aac10"
format_stamp: "Formatted at 2026-09-29 18:41:28 on dist-test-slave-j1xs"
server_key: "fc16c8901cbec5ff07a71fa8cf17eca1"
server_key_iv: "b9e1bcfb4321aac62b776693eb8b92b4"
server_key_version: "encryptionkey@0"
I20260929 18:41:28.952939 965 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707036041078-13712-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707036041078-13712-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707036041078-13712-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:41:28.963610 965 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:41:28.964210 965 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:41:28.964394 965 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:41:28.971216 965 rpc_server.cc:307] RPC server started. Bound to: 127.13.100.62:37343
I20260929 18:41:28.971227 1031 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.100.62:37343 every 8 connection(s)
I20260929 18:41:28.972061 965 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
I20260929 18:41:28.974771 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 965
I20260929 18:41:28.974903 13712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707036041078-13712-0/minicluster-data/master-0/wal/instance
I20260929 18:41:28.975210 1032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:41:28.975234 13712 external_mini_cluster.cc:1428] Setting key d63ce2ba3694efd52d8d3582e53dc68b
I20260929 18:41:28.982493 1032 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 874a58551e194bbdb820df39e57aac10: Bootstrap starting.
I20260929 18:41:28.984387 1035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:28.976467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54202 (local address 127.13.100.62:37343)
0929 18:41:28.977036 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0929 18:41:28.977051 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 18:41:28.977085 (+ 34us) server_negotiation.cc:407] Connection header received
0929 18:41:28.977714 (+ 629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:41:28.977729 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:41:28.978216 (+ 487us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:41:28.978633 (+ 417us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:41:28.979095 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:28.980024 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:28.981232 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:28.981685 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:28.981796 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:28.982730 (+ 934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:41:28.982736 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:41:28.982747 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:41:28.983118 (+ 371us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:41:28.983167 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:41:28.983186 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:41:28.983261 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 18:41:28.983426 (+ 165us) server_negotiation.cc:299] Negotiation successful
0929 18:41:28.983698 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":111,"threads_started":1}
I20260929 18:41:28.984794 1032 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 874a58551e194bbdb820df39e57aac10: Neither blocks nor log segments found. Creating new log.
I20260929 18:41:28.985556 1032 log.cc:824] T 00000000000000000000000000000000 P 874a58551e194bbdb820df39e57aac10: Log is configured to *not* fsync() on all Append() calls
I20260929 18:41:28.987429 1032 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 874a58551e194bbdb820df39e57aac10: No bootstrap required, opened a new log
I20260929 18:41:28.990840 1032 raft_consensus.cc:348] T 00000000000000000000000000000000 P 874a58551e194bbdb820df39e57aac10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "874a58551e194bbdb820df39e57aac10" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 37343 } }
I20260929 18:41:28.991045 1032 raft_consensus.cc:374] T 00000000000000000000000000000000 P 874a58551e194bbdb820df39e57aac10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:41:28.991079 1032 raft_consensus.cc:729] T 00000000000000000000000000000000 P 874a58551e194bbdb820df39e57aac10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 874a58551e194bbdb820df39e57aac10, State: Initialized, Role: FOLLOWER
I20260929 18:41:28.991420 1032 consensus_queue.cc:260] T 00000000000000000000000000000000 P 874a58551e194bbdb820df39e57aac10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "874a58551e194bbdb820df39e57aac10" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 37343 } }
I20260929 18:41:28.991523 1032 raft_consensus.cc:388] T 00000000000000000000000000000000 P 874a58551e194bbdb820df39e57aac10 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:41:28.991560 1032 raft_consensus.cc:482] T 00000000000000000000000000000000 P 874a58551e194bbdb820df39e57aac10 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:41:28.991626 1032 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 874a58551e194bbdb820df39e57aac10 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:41:28.992432 1032 raft_consensus.cc:504] T 00000000000000000000000000000000 P 874a58551e194bbdb820df39e57aac10 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "874a58551e194bbdb820df39e57aac10" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 37343 } }
I20260929 18:41:28.992712 1032 leader_election.cc:302] T 00000000000000000000000000000000 P 874a58551e194bbdb820df39e57aac10 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 874a58551e194bbdb820df39e57aac10; no voters:
I20260929 18:41:28.992962 1032 leader_election.cc:288] T 00000000000000000000000000000000 P 874a58551e194bbdb820df39e57aac10 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:41:28.993117 1037 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 874a58551e194bbdb820df39e57aac10 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:41:28.993368 1037 raft_consensus.cc:686] T 00000000000000000000000000000000 P 874a58551e194bbdb820df39e57aac10 [term 1 LEADER]: Becoming Leader. State: Replica: 874a58551e194bbdb820df39e57aac10, State: Running, Role: LEADER
I20260929 18:41:28.993681 1037 consensus_queue.cc:237] T 00000000000000000000000000000000 P 874a58551e194bbdb820df39e57aac10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "874a58551e194bbdb820df39e57aac10" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 37343 } }
I20260929 18:41:28.994187 1032 sys_catalog.cc:564] T 00000000000000000000000000000000 P 874a58551e194bbdb820df39e57aac10 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:41:28.995551 1039 sys_catalog.cc:455] T 00000000000000000000000000000000 P 874a58551e194bbdb820df39e57aac10 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 874a58551e194bbdb820df39e57aac10. Latest consensus state: current_term: 1 leader_uuid: "874a58551e194bbdb820df39e57aac10" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "874a58551e194bbdb820df39e57aac10" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 37343 } } }
I20260929 18:41:28.995872 1039 sys_catalog.cc:458] T 00000000000000000000000000000000 P 874a58551e194bbdb820df39e57aac10 [sys.catalog]: This master's current role is: LEADER
I20260929 18:41:28.995842 1038 sys_catalog.cc:455] T 00000000000000000000000000000000 P 874a58551e194bbdb820df39e57aac10 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "874a58551e194bbdb820df39e57aac10" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "874a58551e194bbdb820df39e57aac10" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 37343 } } }
I20260929 18:41:28.995980 1038 sys_catalog.cc:458] T 00000000000000000000000000000000 P 874a58551e194bbdb820df39e57aac10 [sys.catalog]: This master's current role is: LEADER
I20260929 18:41:28.996731 1048 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:41:28.999358 1048 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:41:29.004864 1048 catalog_manager.cc:1347] Generated new cluster ID: 756993e0810f4d4d80689ca8e9da03b2
I20260929 18:41:29.004928 1048 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:41:29.018515 1048 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:41:29.019392 1048 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:41:29.027180 1048 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 874a58551e194bbdb820df39e57aac10: Generated new TSK 0
I20260929 18:41:29.027782 1048 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:41:29.032920 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707036041078-13712-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707036041078-13712-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707036041078-13712-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707036041078-13712-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.100.1:0
--local_ip_for_outbound_sockets=127.13.100.1
--webserver_interface=127.13.100.1
--webserver_port=0
--tserver_master_addrs=127.13.100.62:37343
--builtin_ntp_servers=127.13.100.20:39531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 18:41:29.148800 1056 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:41:29.149060 1056 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:41:29.149099 1056 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:41:29.153182 1056 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:41:29.153241 1056 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:41:29.153335 1056 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.100.1
I20260929 18:41:29.158392 1056 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.100.20:39531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707036041078-13712-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707036041078-13712-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.100.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707036041078-13712-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.100.1
--webserver_port=0
--tserver_master_addrs=127.13.100.62:37343
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1056
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.100.1
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707036041078-13712-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:41:29.159543 1056 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:41:29.160931 1056 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:41:29.170418 1062 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:41:29.170457 1061 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:41:29.170457 1064 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:41:29.170674 1056 server_base.cc:1034] running on GCE node
I20260929 18:41:29.171152 1056 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:41:29.171818 1056 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:41:29.173019 1056 hybrid_clock.cc:648] HybridClock initialized: now 1790707289173003 us; error 34 us; skew 500 ppm
I20260929 18:41:29.174861 1056 webserver.cc:460] Webserver started at http://127.13.100.1:46549/ using document root <none> and password file <none>
I20260929 18:41:29.175418 1056 fs_manager.cc:359] Metadata directory not provided
I20260929 18:41:29.175501 1056 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:41:29.175720 1056 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:41:29.177670 1056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707036041078-13712-0/minicluster-data/ts-0/data/instance:
uuid: "b987e98b2a834331af07fde1482e62f0"
format_stamp: "Formatted at 2026-09-29 18:41:29 on dist-test-slave-j1xs"
server_key: "9f12d68eeb63989346a9402cd965d8ae"
server_key_iv: "04e734a41b146a830e7b8287aa3d936a"
server_key_version: "encryptionkey@0"
I20260929 18:41:29.178265 1056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707036041078-13712-0/minicluster-data/ts-0/wal/instance:
uuid: "b987e98b2a834331af07fde1482e62f0"
format_stamp: "Formatted at 2026-09-29 18:41:29 on dist-test-slave-j1xs"
server_key: "9f12d68eeb63989346a9402cd965d8ae"
server_key_iv: "04e734a41b146a830e7b8287aa3d936a"
server_key_version: "encryptionkey@0"
I20260929 18:41:29.181845 1056 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:41:29.184190 1070 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:41:29.185355 1056 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:41:29.185479 1056 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707036041078-13712-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707036041078-13712-0/minicluster-data/ts-0/wal
uuid: "b987e98b2a834331af07fde1482e62f0"
format_stamp: "Formatted at 2026-09-29 18:41:29 on dist-test-slave-j1xs"
server_key: "9f12d68eeb63989346a9402cd965d8ae"
server_key_iv: "04e734a41b146a830e7b8287aa3d936a"
server_key_version: "encryptionkey@0"
I20260929 18:41:29.185583 1056 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707036041078-13712-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707036041078-13712-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707036041078-13712-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:41:29.195817 1056 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:41:29.196411 1056 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:41:29.196583 1056 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:41:29.197116 1056 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:41:29.198033 1056 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:41:29.198081 1056 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:41:29.198118 1056 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:41:29.198133 1056 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:41:29.206624 1056 rpc_server.cc:307] RPC server started. Bound to: 127.13.100.1:40035
I20260929 18:41:29.206717 1183 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.100.1:40035 every 8 connection(s)
I20260929 18:41:29.207523 1056 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707036041078-13712-0/minicluster-data/ts-0/data/info.pb
I20260929 18:41:29.209105 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 1056
I20260929 18:41:29.209244 13712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707036041078-13712-0/minicluster-data/ts-0/wal/instance
I20260929 18:41:29.209569 13712 external_mini_cluster.cc:1428] Setting key b538fca4c149b2b96c836a06f34ff284
I20260929 18:41:29.211488 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707036041078-13712-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707036041078-13712-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707036041078-13712-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707036041078-13712-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.100.2:0
--local_ip_for_outbound_sockets=127.13.100.2
--webserver_interface=127.13.100.2
--webserver_port=0
--tserver_master_addrs=127.13.100.62:37343
--builtin_ntp_servers=127.13.100.20:39531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 18:41:29.217536 1035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:29.209309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.1:53555 (local address 127.13.100.62:37343)
0929 18:41:29.209464 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0929 18:41:29.209468 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:41:29.210606 (+ 1138us) server_negotiation.cc:407] Connection header received
0929 18:41:29.212074 (+ 1468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:41:29.212080 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:41:29.212142 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:41:29.212255 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:41:29.213123 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:29.214092 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:29.215226 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:29.215479 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:29.215589 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:29.217039 (+ 1450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:41:29.217043 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:41:29.217045 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:41:29.217056 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:41:29.217076 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:41:29.217084 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:41:29.217148 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0929 18:41:29.217364 (+ 216us) server_negotiation.cc:299] Negotiation successful
0929 18:41:29.217423 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 18:41:29.218844 1186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:29.209747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:37343 (local address 127.13.100.1:53555)
0929 18:41:29.210450 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:29.210501 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 18:41:29.211839 (+ 1338us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:41:29.212399 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:41:29.212405 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:41:29.212483 (+ 78us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:41:29.212980 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:41:29.212993 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:41:29.214216 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:41:29.214219 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:41:29.215096 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:41:29.215106 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:41:29.215657 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:41:29.215664 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:41:29.215917 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:29.216695 (+ 778us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:41:29.216699 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:41:29.216756 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:41:29.216850 (+ 94us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:41:29.217175 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:41:29.217186 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:41:29.217196 (+ 10us) client_negotiation.cc:769] Sending connection context
0929 18:41:29.217362 (+ 166us) client_negotiation.cc:240] Negotiation successful
0929 18:41:29.217690 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":139,"threads_started":1}
I20260929 18:41:29.220320 1184 heartbeater.cc:347] Connected to a master server at 127.13.100.62:37343
I20260929 18:41:29.220570 1184 heartbeater.cc:464] Registering TS with master...
I20260929 18:41:29.221094 1184 heartbeater.cc:511] Master 127.13.100.62:37343 requested a full tablet report, sending...
I20260929 18:41:29.222759 996 ts_manager.cc:194] Registered new tserver with Master: b987e98b2a834331af07fde1482e62f0 (127.13.100.1:40035)
I20260929 18:41:29.224310 996 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.100.1:53555
W20260929 18:41:29.326092 1187 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:41:29.326385 1187 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:41:29.326424 1187 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:41:29.330521 1187 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:41:29.330583 1187 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:41:29.330639 1187 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.100.2
I20260929 18:41:29.336017 1187 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.100.20:39531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707036041078-13712-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707036041078-13712-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.100.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707036041078-13712-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.100.2
--webserver_port=0
--tserver_master_addrs=127.13.100.62:37343
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1187
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.100.2
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707036041078-13712-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:41:29.337512 1187 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:41:29.339148 1187 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:41:29.349248 1195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:41:29.349294 1192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:41:29.349311 1193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:41:29.349677 1187 server_base.cc:1034] running on GCE node
I20260929 18:41:29.350193 1187 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:41:29.350813 1187 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:41:29.352025 1187 hybrid_clock.cc:648] HybridClock initialized: now 1790707289352012 us; error 27 us; skew 500 ppm
I20260929 18:41:29.354089 1187 webserver.cc:460] Webserver started at http://127.13.100.2:34163/ using document root <none> and password file <none>
I20260929 18:41:29.354671 1187 fs_manager.cc:359] Metadata directory not provided
I20260929 18:41:29.354769 1187 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:41:29.355000 1187 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:41:29.357180 1187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707036041078-13712-0/minicluster-data/ts-1/data/instance:
uuid: "cfada51e9a7d4c359e1564c522b80ee4"
format_stamp: "Formatted at 2026-09-29 18:41:29 on dist-test-slave-j1xs"
server_key: "00caf32d978e1e48b591c10b2606a933"
server_key_iv: "0bddba9d4e7c883608400522b68c0666"
server_key_version: "encryptionkey@0"
I20260929 18:41:29.358210 1187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707036041078-13712-0/minicluster-data/ts-1/wal/instance:
uuid: "cfada51e9a7d4c359e1564c522b80ee4"
format_stamp: "Formatted at 2026-09-29 18:41:29 on dist-test-slave-j1xs"
server_key: "00caf32d978e1e48b591c10b2606a933"
server_key_iv: "0bddba9d4e7c883608400522b68c0666"
server_key_version: "encryptionkey@0"
I20260929 18:41:29.362008 1187 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 18:41:29.364460 1201 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:41:29.365821 1187 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:41:29.365991 1187 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707036041078-13712-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707036041078-13712-0/minicluster-data/ts-1/wal
uuid: "cfada51e9a7d4c359e1564c522b80ee4"
format_stamp: "Formatted at 2026-09-29 18:41:29 on dist-test-slave-j1xs"
server_key: "00caf32d978e1e48b591c10b2606a933"
server_key_iv: "0bddba9d4e7c883608400522b68c0666"
server_key_version: "encryptionkey@0"
I20260929 18:41:29.366128 1187 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707036041078-13712-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707036041078-13712-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707036041078-13712-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:41:29.382534 1187 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:41:29.383211 1187 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:41:29.383410 1187 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:41:29.383983 1187 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:41:29.384927 1187 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:41:29.384994 1187 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:41:29.385058 1187 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:41:29.385102 1187 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:41:29.393843 1187 rpc_server.cc:307] RPC server started. Bound to: 127.13.100.2:35927
I20260929 18:41:29.393879 1314 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.100.2:35927 every 8 connection(s)
I20260929 18:41:29.394814 1187 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707036041078-13712-0/minicluster-data/ts-1/data/info.pb
I20260929 18:41:29.398478 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 1187
I20260929 18:41:29.398602 13712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707036041078-13712-0/minicluster-data/ts-1/wal/instance
I20260929 18:41:29.398924 13712 external_mini_cluster.cc:1428] Setting key 2ae0d907bda434629fbbeb210c2c8319
I20260929 18:41:29.401053 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707036041078-13712-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707036041078-13712-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707036041078-13712-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707036041078-13712-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.100.3:0
--local_ip_for_outbound_sockets=127.13.100.3
--webserver_interface=127.13.100.3
--webserver_port=0
--tserver_master_addrs=127.13.100.62:37343
--builtin_ntp_servers=127.13.100.20:39531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 18:41:29.405309 1035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:29.396664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.2:53437 (local address 127.13.100.62:37343)
0929 18:41:29.396858 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0929 18:41:29.396862 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:41:29.397918 (+ 1056us) server_negotiation.cc:407] Connection header received
0929 18:41:29.399426 (+ 1508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:41:29.399429 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:41:29.399496 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:41:29.399582 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:41:29.400729 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:29.401534 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:29.403230 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:29.403447 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:29.403531 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:29.404669 (+ 1138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:41:29.404676 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:41:29.404683 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:41:29.404700 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:41:29.404726 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:41:29.404739 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:41:29.404826 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 18:41:29.404976 (+ 150us) server_negotiation.cc:299] Negotiation successful
0929 18:41:29.405055 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260929 18:41:29.406131 1317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:29.397075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:37343 (local address 127.13.100.2:53437)
0929 18:41:29.397711 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:29.397749 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 18:41:29.399190 (+ 1441us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:41:29.399817 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:41:29.399825 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:41:29.399927 (+ 102us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:41:29.400573 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:41:29.400590 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:41:29.401660 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:41:29.401663 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:41:29.403097 (+ 1434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:41:29.403111 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:41:29.403587 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:41:29.403590 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:41:29.403756 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:29.404414 (+ 658us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:41:29.404415 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:41:29.404453 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:41:29.404514 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:41:29.404841 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:41:29.404847 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:41:29.404851 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 18:41:29.404945 (+ 94us) client_negotiation.cc:240] Negotiation successful
0929 18:41:29.405238 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":118,"threads_started":1}
I20260929 18:41:29.407356 1315 heartbeater.cc:347] Connected to a master server at 127.13.100.62:37343
I20260929 18:41:29.407588 1315 heartbeater.cc:464] Registering TS with master...
I20260929 18:41:29.408102 1315 heartbeater.cc:511] Master 127.13.100.62:37343 requested a full tablet report, sending...
I20260929 18:41:29.409153 996 ts_manager.cc:194] Registered new tserver with Master: cfada51e9a7d4c359e1564c522b80ee4 (127.13.100.2:35927)
I20260929 18:41:29.409776 996 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.100.2:53437
W20260929 18:41:29.512152 1318 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:41:29.512415 1318 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:41:29.512454 1318 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:41:29.516491 1318 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:41:29.516546 1318 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:41:29.516601 1318 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.100.3
I20260929 18:41:29.521363 1318 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.100.20:39531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707036041078-13712-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707036041078-13712-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.100.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707036041078-13712-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.100.3
--webserver_port=0
--tserver_master_addrs=127.13.100.62:37343
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1318
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.100.3
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707036041078-13712-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:41:29.522522 1318 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:41:29.523940 1318 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 18:41:29.534192 1318 server_base.cc:1034] running on GCE node
W20260929 18:41:29.534137 1323 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:41:29.534111 1326 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:41:29.534137 1324 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:41:29.534747 1318 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:41:29.535355 1318 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:41:29.536521 1318 hybrid_clock.cc:648] HybridClock initialized: now 1790707289536485 us; error 48 us; skew 500 ppm
I20260929 18:41:29.538371 1318 webserver.cc:460] Webserver started at http://127.13.100.3:37005/ using document root <none> and password file <none>
I20260929 18:41:29.538935 1318 fs_manager.cc:359] Metadata directory not provided
I20260929 18:41:29.539012 1318 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:41:29.539264 1318 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:41:29.541245 1318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707036041078-13712-0/minicluster-data/ts-2/data/instance:
uuid: "db313c76d8d64e5b8aa8e0a2cdf8ee38"
format_stamp: "Formatted at 2026-09-29 18:41:29 on dist-test-slave-j1xs"
server_key: "f1710c306a224d96665f070648936612"
server_key_iv: "da6944f4c296e6aa44f5cc9b414dc40d"
server_key_version: "encryptionkey@0"
I20260929 18:41:29.541795 1318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707036041078-13712-0/minicluster-data/ts-2/wal/instance:
uuid: "db313c76d8d64e5b8aa8e0a2cdf8ee38"
format_stamp: "Formatted at 2026-09-29 18:41:29 on dist-test-slave-j1xs"
server_key: "f1710c306a224d96665f070648936612"
server_key_iv: "da6944f4c296e6aa44f5cc9b414dc40d"
server_key_version: "encryptionkey@0"
I20260929 18:41:29.545455 1318 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:41:29.547897 1332 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:41:29.549171 1318 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:41:29.549327 1318 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707036041078-13712-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707036041078-13712-0/minicluster-data/ts-2/wal
uuid: "db313c76d8d64e5b8aa8e0a2cdf8ee38"
format_stamp: "Formatted at 2026-09-29 18:41:29 on dist-test-slave-j1xs"
server_key: "f1710c306a224d96665f070648936612"
server_key_iv: "da6944f4c296e6aa44f5cc9b414dc40d"
server_key_version: "encryptionkey@0"
I20260929 18:41:29.549448 1318 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707036041078-13712-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707036041078-13712-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707036041078-13712-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:41:29.562211 1318 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:41:29.562865 1318 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:41:29.563071 1318 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:41:29.563647 1318 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:41:29.564617 1318 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:41:29.564698 1318 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:41:29.564774 1318 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:41:29.564826 1318 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:41:29.574663 1318 rpc_server.cc:307] RPC server started. Bound to: 127.13.100.3:35897
I20260929 18:41:29.574681 1445 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.100.3:35897 every 8 connection(s)
I20260929 18:41:29.575642 1318 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707036041078-13712-0/minicluster-data/ts-2/data/info.pb
I20260929 18:41:29.577943 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 1318
I20260929 18:41:29.578063 13712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707036041078-13712-0/minicluster-data/ts-2/wal/instance
I20260929 18:41:29.578426 13712 external_mini_cluster.cc:1428] Setting key db5b261a400867bc4c752d2c62b94c38
I20260929 18:41:29.585552 1035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:29.577493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.3:51589 (local address 127.13.100.62:37343)
0929 18:41:29.577632 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0929 18:41:29.577636 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:41:29.578950 (+ 1314us) server_negotiation.cc:407] Connection header received
0929 18:41:29.580765 (+ 1815us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:41:29.580768 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:41:29.580820 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:41:29.580897 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:41:29.581793 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:29.582411 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:29.583541 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:29.583786 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:29.583896 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:29.585019 (+ 1123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:41:29.585023 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:41:29.585026 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:41:29.585040 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:41:29.585067 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:41:29.585075 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:41:29.585170 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 18:41:29.585335 (+ 165us) server_negotiation.cc:299] Negotiation successful
0929 18:41:29.585401 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 18:41:29.586520 1448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:29.577861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:37343 (local address 127.13.100.3:51589)
0929 18:41:29.578799 (+ 938us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:29.578841 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 18:41:29.580548 (+ 1707us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:41:29.581071 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:41:29.581078 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:41:29.581165 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:41:29.581650 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:41:29.581662 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:41:29.582533 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:41:29.582536 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:41:29.583419 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:41:29.583429 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:41:29.583935 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:41:29.583939 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:41:29.584115 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:29.584706 (+ 591us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:41:29.584707 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:41:29.584798 (+ 91us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:41:29.584867 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:41:29.585170 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:41:29.585182 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:41:29.585190 (+ 8us) client_negotiation.cc:769] Sending connection context
0929 18:41:29.585324 (+ 134us) client_negotiation.cc:240] Negotiation successful
0929 18:41:29.585626 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":156,"threads_started":1}
I20260929 18:41:29.587668 1446 heartbeater.cc:347] Connected to a master server at 127.13.100.62:37343
I20260929 18:41:29.587917 1446 heartbeater.cc:464] Registering TS with master...
I20260929 18:41:29.588666 1446 heartbeater.cc:511] Master 127.13.100.62:37343 requested a full tablet report, sending...
I20260929 18:41:29.589814 996 ts_manager.cc:194] Registered new tserver with Master: db313c76d8d64e5b8aa8e0a2cdf8ee38 (127.13.100.3:35897)
I20260929 18:41:29.590495 996 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.100.3:51589
I20260929 18:41:29.592566 13712 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 18:41:29.598290 1035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:29.594601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54214 (local address 127.13.100.62:37343)
0929 18:41:29.594725 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0929 18:41:29.594730 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:41:29.594844 (+ 114us) server_negotiation.cc:407] Connection header received
0929 18:41:29.594933 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:41:29.594936 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:41:29.594971 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:41:29.595039 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:41:29.595333 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:29.595876 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:29.596744 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:29.596982 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:29.597070 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:29.597861 (+ 791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:41:29.597864 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:41:29.597866 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:41:29.597877 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:41:29.597948 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:41:29.597965 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:41:29.598027 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0929 18:41:29.598129 (+ 102us) server_negotiation.cc:299] Negotiation successful
0929 18:41:29.598174 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 18:41:29.600875 996 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54214:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:41:29.603888 996 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:41:29.617880 1458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:29.612109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33534 (local address 127.13.100.3:35897)
0929 18:41:29.612522 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0929 18:41:29.612534 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 18:41:29.612561 (+ 27us) server_negotiation.cc:407] Connection header received
0929 18:41:29.612639 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:41:29.612645 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:41:29.612867 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:41:29.613020 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:41:29.613504 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:29.614668 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:29.616308 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:29.617102 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:29.617235 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:29.617435 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0929 18:41:29.617592 (+ 157us) server_negotiation.cc:299] Negotiation successful
0929 18:41:29.617704 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":135,"threads_started":1}
I20260929 18:41:29.618011 1459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:29.612966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.1:40035 (local address 127.0.0.1:57372)
0929 18:41:29.613404 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:29.613419 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 18:41:29.613493 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:41:29.614192 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:41:29.614194 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:41:29.614212 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:41:29.614307 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:41:29.614312 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:41:29.615606 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:41:29.615609 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:41:29.616619 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:41:29.616625 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:41:29.617409 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:41:29.617412 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:41:29.617481 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:29.617500 (+ 19us) client_negotiation.cc:769] Sending connection context
0929 18:41:29.617547 (+ 47us) client_negotiation.cc:240] Negotiation successful
0929 18:41:29.617760 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":376,"mutex_wait_us":21,"thread_start_us":89,"threads_started":1}
I20260929 18:41:29.618124 1457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:29.611924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.3:35897 (local address 127.0.0.1:33534)
0929 18:41:29.612278 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:29.612339 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0929 18:41:29.612469 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:41:29.613198 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:41:29.613204 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:41:29.613225 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:41:29.613366 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:41:29.613374 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:41:29.614790 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:41:29.614793 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:41:29.616178 (+ 1385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:41:29.616187 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:41:29.617240 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:41:29.617244 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:41:29.617395 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:29.617448 (+ 53us) client_negotiation.cc:769] Sending connection context
0929 18:41:29.617554 (+ 106us) client_negotiation.cc:240] Negotiation successful
0929 18:41:29.617843 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"mutex_wait_us":79,"thread_start_us":110,"threads_started":1}
I20260929 18:41:29.618264 1461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:29.612659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.2:35927 (local address 127.0.0.1:56894)
0929 18:41:29.613203 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:29.613214 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 18:41:29.613324 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:41:29.613990 (+ 666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:41:29.613993 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:41:29.614010 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:41:29.614110 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:41:29.614119 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:41:29.615138 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:41:29.615142 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:41:29.616328 (+ 1186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:41:29.616338 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:41:29.617500 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:41:29.617504 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:41:29.617615 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:29.617637 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 18:41:29.617709 (+ 72us) client_negotiation.cc:240] Negotiation successful
0929 18:41:29.617792 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"mutex_wait_us":3,"thread_start_us":83,"threads_started":1}
I20260929 18:41:29.618471 1460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:29.612824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56894 (local address 127.13.100.2:35927)
0929 18:41:29.613563 (+ 739us) server_negotiation.cc:206] Beginning negotiation
0929 18:41:29.613574 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 18:41:29.613595 (+ 21us) server_negotiation.cc:407] Connection header received
0929 18:41:29.613668 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:41:29.613673 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:41:29.613817 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:41:29.614009 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:41:29.614205 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:29.615021 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:29.616449 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:29.617165 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:29.617254 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:29.617435 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0929 18:41:29.618191 (+ 756us) server_negotiation.cc:299] Negotiation successful
0929 18:41:29.618301 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":628,"thread_start_us":120,"threads_started":1}
I20260929 18:41:29.618826 1462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:29.613379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57372 (local address 127.13.100.1:40035)
0929 18:41:29.613738 (+ 359us) server_negotiation.cc:206] Beginning negotiation
0929 18:41:29.613749 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 18:41:29.613770 (+ 21us) server_negotiation.cc:407] Connection header received
0929 18:41:29.613842 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:41:29.613846 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:41:29.614045 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:41:29.614434 (+ 389us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:41:29.614554 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:29.615480 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:29.616733 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:29.617266 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:29.617792 (+ 526us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:29.618551 (+ 759us) server_negotiation.cc:1035] Waiting for connection context
0929 18:41:29.618610 (+ 59us) server_negotiation.cc:299] Negotiation successful
0929 18:41:29.618704 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":136,"threads_started":1}
I20260929 18:41:29.621124 1118 tablet_service.cc:1467] Processing CreateTablet for tablet 997d78333451488ba6063d7c596cf93d (DEFAULT_TABLE table=test-table [id=dbaaf3fd4f834672b9881db236591ddb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:41:29.621160 1249 tablet_service.cc:1467] Processing CreateTablet for tablet 997d78333451488ba6063d7c596cf93d (DEFAULT_TABLE table=test-table [id=dbaaf3fd4f834672b9881db236591ddb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:41:29.621160 1380 tablet_service.cc:1467] Processing CreateTablet for tablet 997d78333451488ba6063d7c596cf93d (DEFAULT_TABLE table=test-table [id=dbaaf3fd4f834672b9881db236591ddb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:41:29.622138 1118 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 997d78333451488ba6063d7c596cf93d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:41:29.622181 1380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 997d78333451488ba6063d7c596cf93d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:41:29.622181 1249 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 997d78333451488ba6063d7c596cf93d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:41:29.628681 1463 tablet_bootstrap.cc:492] T 997d78333451488ba6063d7c596cf93d P b987e98b2a834331af07fde1482e62f0: Bootstrap starting.
I20260929 18:41:29.630038 1465 tablet_bootstrap.cc:492] T 997d78333451488ba6063d7c596cf93d P db313c76d8d64e5b8aa8e0a2cdf8ee38: Bootstrap starting.
I20260929 18:41:29.631140 1463 tablet_bootstrap.cc:654] T 997d78333451488ba6063d7c596cf93d P b987e98b2a834331af07fde1482e62f0: Neither blocks nor log segments found. Creating new log.
I20260929 18:41:29.631987 1463 log.cc:824] T 997d78333451488ba6063d7c596cf93d P b987e98b2a834331af07fde1482e62f0: Log is configured to *not* fsync() on all Append() calls
I20260929 18:41:29.632135 1465 tablet_bootstrap.cc:654] T 997d78333451488ba6063d7c596cf93d P db313c76d8d64e5b8aa8e0a2cdf8ee38: Neither blocks nor log segments found. Creating new log.
I20260929 18:41:29.632385 1464 tablet_bootstrap.cc:492] T 997d78333451488ba6063d7c596cf93d P cfada51e9a7d4c359e1564c522b80ee4: Bootstrap starting.
I20260929 18:41:29.632927 1465 log.cc:824] T 997d78333451488ba6063d7c596cf93d P db313c76d8d64e5b8aa8e0a2cdf8ee38: Log is configured to *not* fsync() on all Append() calls
I20260929 18:41:29.634402 1464 tablet_bootstrap.cc:654] T 997d78333451488ba6063d7c596cf93d P cfada51e9a7d4c359e1564c522b80ee4: Neither blocks nor log segments found. Creating new log.
I20260929 18:41:29.634681 1463 tablet_bootstrap.cc:492] T 997d78333451488ba6063d7c596cf93d P b987e98b2a834331af07fde1482e62f0: No bootstrap required, opened a new log
I20260929 18:41:29.634848 1465 tablet_bootstrap.cc:492] T 997d78333451488ba6063d7c596cf93d P db313c76d8d64e5b8aa8e0a2cdf8ee38: No bootstrap required, opened a new log
I20260929 18:41:29.634886 1463 ts_tablet_manager.cc:1397] T 997d78333451488ba6063d7c596cf93d P b987e98b2a834331af07fde1482e62f0: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260929 18:41:29.635007 1465 ts_tablet_manager.cc:1397] T 997d78333451488ba6063d7c596cf93d P db313c76d8d64e5b8aa8e0a2cdf8ee38: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 18:41:29.635134 1464 log.cc:824] T 997d78333451488ba6063d7c596cf93d P cfada51e9a7d4c359e1564c522b80ee4: Log is configured to *not* fsync() on all Append() calls
I20260929 18:41:29.636610 1464 tablet_bootstrap.cc:492] T 997d78333451488ba6063d7c596cf93d P cfada51e9a7d4c359e1564c522b80ee4: No bootstrap required, opened a new log
I20260929 18:41:29.636801 1464 ts_tablet_manager.cc:1397] T 997d78333451488ba6063d7c596cf93d P cfada51e9a7d4c359e1564c522b80ee4: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20260929 18:41:29.638880 1463 raft_consensus.cc:348] T 997d78333451488ba6063d7c596cf93d P b987e98b2a834331af07fde1482e62f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db313c76d8d64e5b8aa8e0a2cdf8ee38" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 35897 } } peers { permanent_uuid: "cfada51e9a7d4c359e1564c522b80ee4" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 35927 } } peers { permanent_uuid: "b987e98b2a834331af07fde1482e62f0" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 40035 } }
I20260929 18:41:29.638952 1465 raft_consensus.cc:348] T 997d78333451488ba6063d7c596cf93d P db313c76d8d64e5b8aa8e0a2cdf8ee38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db313c76d8d64e5b8aa8e0a2cdf8ee38" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 35897 } } peers { permanent_uuid: "cfada51e9a7d4c359e1564c522b80ee4" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 35927 } } peers { permanent_uuid: "b987e98b2a834331af07fde1482e62f0" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 40035 } }
I20260929 18:41:29.639151 1463 raft_consensus.cc:374] T 997d78333451488ba6063d7c596cf93d P b987e98b2a834331af07fde1482e62f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:41:29.639151 1465 raft_consensus.cc:374] T 997d78333451488ba6063d7c596cf93d P db313c76d8d64e5b8aa8e0a2cdf8ee38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:41:29.639214 1465 raft_consensus.cc:729] T 997d78333451488ba6063d7c596cf93d P db313c76d8d64e5b8aa8e0a2cdf8ee38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db313c76d8d64e5b8aa8e0a2cdf8ee38, State: Initialized, Role: FOLLOWER
I20260929 18:41:29.639215 1463 raft_consensus.cc:729] T 997d78333451488ba6063d7c596cf93d P b987e98b2a834331af07fde1482e62f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b987e98b2a834331af07fde1482e62f0, State: Initialized, Role: FOLLOWER
I20260929 18:41:29.639662 1463 consensus_queue.cc:260] T 997d78333451488ba6063d7c596cf93d P b987e98b2a834331af07fde1482e62f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db313c76d8d64e5b8aa8e0a2cdf8ee38" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 35897 } } peers { permanent_uuid: "cfada51e9a7d4c359e1564c522b80ee4" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 35927 } } peers { permanent_uuid: "b987e98b2a834331af07fde1482e62f0" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 40035 } }
I20260929 18:41:29.639662 1465 consensus_queue.cc:260] T 997d78333451488ba6063d7c596cf93d P db313c76d8d64e5b8aa8e0a2cdf8ee38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db313c76d8d64e5b8aa8e0a2cdf8ee38" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 35897 } } peers { permanent_uuid: "cfada51e9a7d4c359e1564c522b80ee4" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 35927 } } peers { permanent_uuid: "b987e98b2a834331af07fde1482e62f0" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 40035 } }
I20260929 18:41:29.639983 1464 raft_consensus.cc:348] T 997d78333451488ba6063d7c596cf93d P cfada51e9a7d4c359e1564c522b80ee4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db313c76d8d64e5b8aa8e0a2cdf8ee38" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 35897 } } peers { permanent_uuid: "cfada51e9a7d4c359e1564c522b80ee4" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 35927 } } peers { permanent_uuid: "b987e98b2a834331af07fde1482e62f0" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 40035 } }
I20260929 18:41:29.640205 1464 raft_consensus.cc:374] T 997d78333451488ba6063d7c596cf93d P cfada51e9a7d4c359e1564c522b80ee4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:41:29.640264 1464 raft_consensus.cc:729] T 997d78333451488ba6063d7c596cf93d P cfada51e9a7d4c359e1564c522b80ee4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfada51e9a7d4c359e1564c522b80ee4, State: Initialized, Role: FOLLOWER
I20260929 18:41:29.640653 1465 ts_tablet_manager.cc:1428] T 997d78333451488ba6063d7c596cf93d P db313c76d8d64e5b8aa8e0a2cdf8ee38: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 18:41:29.640653 1463 ts_tablet_manager.cc:1428] T 997d78333451488ba6063d7c596cf93d P b987e98b2a834331af07fde1482e62f0: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 18:41:29.640702 1464 consensus_queue.cc:260] T 997d78333451488ba6063d7c596cf93d P cfada51e9a7d4c359e1564c522b80ee4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db313c76d8d64e5b8aa8e0a2cdf8ee38" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 35897 } } peers { permanent_uuid: "cfada51e9a7d4c359e1564c522b80ee4" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 35927 } } peers { permanent_uuid: "b987e98b2a834331af07fde1482e62f0" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 40035 } }
I20260929 18:41:29.640969 1446 heartbeater.cc:503] Master 127.13.100.62:37343 was elected leader, sending a full tablet report...
I20260929 18:41:29.641414 1184 heartbeater.cc:503] Master 127.13.100.62:37343 was elected leader, sending a full tablet report...
I20260929 18:41:29.641621 1464 ts_tablet_manager.cc:1428] T 997d78333451488ba6063d7c596cf93d P cfada51e9a7d4c359e1564c522b80ee4: Time spent starting tablet: real 0.005s user 0.001s sys 0.004s
I20260929 18:41:29.641963 1315 heartbeater.cc:503] Master 127.13.100.62:37343 was elected leader, sending a full tablet report...
W20260929 18:41:29.645753 1316 tablet.cc:2369] T 997d78333451488ba6063d7c596cf93d P cfada51e9a7d4c359e1564c522b80ee4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:41:29.676195 1470 raft_consensus.cc:482] T 997d78333451488ba6063d7c596cf93d P db313c76d8d64e5b8aa8e0a2cdf8ee38 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:41:29.676427 1470 raft_consensus.cc:504] T 997d78333451488ba6063d7c596cf93d P db313c76d8d64e5b8aa8e0a2cdf8ee38 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db313c76d8d64e5b8aa8e0a2cdf8ee38" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 35897 } } peers { permanent_uuid: "cfada51e9a7d4c359e1564c522b80ee4" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 35927 } } peers { permanent_uuid: "b987e98b2a834331af07fde1482e62f0" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 40035 } }
I20260929 18:41:29.677551 1470 leader_election.cc:288] T 997d78333451488ba6063d7c596cf93d P db313c76d8d64e5b8aa8e0a2cdf8ee38 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cfada51e9a7d4c359e1564c522b80ee4 (127.13.100.2:35927), b987e98b2a834331af07fde1482e62f0 (127.13.100.1:40035)
I20260929 18:41:29.681797 1448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:29.677876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.2:35927 (local address 127.13.100.3:53515)
0929 18:41:29.678064 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:29.678081 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 18:41:29.678185 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:41:29.678552 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:41:29.678555 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:41:29.678572 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:41:29.678708 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:41:29.678717 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:41:29.679716 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:41:29.679719 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:41:29.680638 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:41:29.680647 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:41:29.681369 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:41:29.681373 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:41:29.681478 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:29.681511 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 18:41:29.681587 (+ 76us) client_negotiation.cc:240] Negotiation successful
0929 18:41:29.681673 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44,"mutex_wait_us":21}
I20260929 18:41:29.681813 1460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:29.678002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.3:53515 (local address 127.13.100.2:35927)
0929 18:41:29.678145 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0929 18:41:29.678149 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:41:29.678164 (+ 15us) server_negotiation.cc:407] Connection header received
0929 18:41:29.678315 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:41:29.678322 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:41:29.678363 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:41:29.678446 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:41:29.678854 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:29.679613 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:29.680775 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:29.681250 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:29.681343 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:29.681380 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0929 18:41:29.681617 (+ 237us) server_negotiation.cc:299] Negotiation successful
0929 18:41:29.681689 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 18:41:29.682139 1472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:29.678219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.1:40035 (local address 127.13.100.3:44497)
0929 18:41:29.678563 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:29.678575 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:41:29.678670 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:41:29.679030 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:41:29.679034 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:41:29.679053 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:41:29.679175 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:41:29.679183 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:41:29.679957 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:41:29.679960 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:41:29.680829 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:41:29.680840 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:41:29.681630 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:41:29.681633 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:41:29.681766 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:29.681788 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 18:41:29.681870 (+ 82us) client_negotiation.cc:240] Negotiation successful
0929 18:41:29.681959 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"mutex_wait_us":1,"thread_start_us":147,"threads_started":1}
I20260929 18:41:29.682451 1462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:29.678354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.100.3:44497 (local address 127.13.100.1:40035)
0929 18:41:29.678497 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0929 18:41:29.678501 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:41:29.678657 (+ 156us) server_negotiation.cc:407] Connection header received
0929 18:41:29.678757 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:41:29.678763 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:41:29.678824 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:41:29.678891 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:41:29.679303 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:29.679831 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:29.680971 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:29.681473 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:29.681603 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:29.681646 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0929 18:41:29.682219 (+ 573us) server_negotiation.cc:299] Negotiation successful
0929 18:41:29.682326 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 18:41:29.682507 1269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997d78333451488ba6063d7c596cf93d" candidate_uuid: "db313c76d8d64e5b8aa8e0a2cdf8ee38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfada51e9a7d4c359e1564c522b80ee4" is_pre_election: true
I20260929 18:41:29.682843 1269 raft_consensus.cc:2454] T 997d78333451488ba6063d7c596cf93d P cfada51e9a7d4c359e1564c522b80ee4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db313c76d8d64e5b8aa8e0a2cdf8ee38 in term 0.
I20260929 18:41:29.683142 1138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997d78333451488ba6063d7c596cf93d" candidate_uuid: "db313c76d8d64e5b8aa8e0a2cdf8ee38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b987e98b2a834331af07fde1482e62f0" is_pre_election: true
I20260929 18:41:29.683359 1335 leader_election.cc:302] T 997d78333451488ba6063d7c596cf93d P db313c76d8d64e5b8aa8e0a2cdf8ee38 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cfada51e9a7d4c359e1564c522b80ee4, db313c76d8d64e5b8aa8e0a2cdf8ee38; no voters:
I20260929 18:41:29.683427 1138 raft_consensus.cc:2454] T 997d78333451488ba6063d7c596cf93d P b987e98b2a834331af07fde1482e62f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db313c76d8d64e5b8aa8e0a2cdf8ee38 in term 0.
I20260929 18:41:29.683622 1470 raft_consensus.cc:2781] T 997d78333451488ba6063d7c596cf93d P db313c76d8d64e5b8aa8e0a2cdf8ee38 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:41:29.683718 1470 raft_consensus.cc:482] T 997d78333451488ba6063d7c596cf93d P db313c76d8d64e5b8aa8e0a2cdf8ee38 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:41:29.683784 1470 raft_consensus.cc:3037] T 997d78333451488ba6063d7c596cf93d P db313c76d8d64e5b8aa8e0a2cdf8ee38 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:41:29.684844 1470 raft_consensus.cc:504] T 997d78333451488ba6063d7c596cf93d P db313c76d8d64e5b8aa8e0a2cdf8ee38 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db313c76d8d64e5b8aa8e0a2cdf8ee38" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 35897 } } peers { permanent_uuid: "cfada51e9a7d4c359e1564c522b80ee4" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 35927 } } peers { permanent_uuid: "b987e98b2a834331af07fde1482e62f0" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 40035 } }
I20260929 18:41:29.685134 1470 leader_election.cc:288] T 997d78333451488ba6063d7c596cf93d P db313c76d8d64e5b8aa8e0a2cdf8ee38 [CANDIDATE]: Term 1 election: Requested vote from peers cfada51e9a7d4c359e1564c522b80ee4 (127.13.100.2:35927), b987e98b2a834331af07fde1482e62f0 (127.13.100.1:40035)
I20260929 18:41:29.685478 1269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997d78333451488ba6063d7c596cf93d" candidate_uuid: "db313c76d8d64e5b8aa8e0a2cdf8ee38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfada51e9a7d4c359e1564c522b80ee4"
I20260929 18:41:29.685518 1138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997d78333451488ba6063d7c596cf93d" candidate_uuid: "db313c76d8d64e5b8aa8e0a2cdf8ee38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b987e98b2a834331af07fde1482e62f0"
I20260929 18:41:29.685621 1269 raft_consensus.cc:3037] T 997d78333451488ba6063d7c596cf93d P cfada51e9a7d4c359e1564c522b80ee4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:41:29.685637 1138 raft_consensus.cc:3037] T 997d78333451488ba6063d7c596cf93d P b987e98b2a834331af07fde1482e62f0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:41:29.686671 1138 raft_consensus.cc:2454] T 997d78333451488ba6063d7c596cf93d P b987e98b2a834331af07fde1482e62f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db313c76d8d64e5b8aa8e0a2cdf8ee38 in term 1.
I20260929 18:41:29.686674 1269 raft_consensus.cc:2454] T 997d78333451488ba6063d7c596cf93d P cfada51e9a7d4c359e1564c522b80ee4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db313c76d8d64e5b8aa8e0a2cdf8ee38 in term 1.
I20260929 18:41:29.686946 1335 leader_election.cc:302] T 997d78333451488ba6063d7c596cf93d P db313c76d8d64e5b8aa8e0a2cdf8ee38 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cfada51e9a7d4c359e1564c522b80ee4, db313c76d8d64e5b8aa8e0a2cdf8ee38; no voters:
I20260929 18:41:29.687139 1470 raft_consensus.cc:2781] T 997d78333451488ba6063d7c596cf93d P db313c76d8d64e5b8aa8e0a2cdf8ee38 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:41:29.687381 1470 raft_consensus.cc:686] T 997d78333451488ba6063d7c596cf93d P db313c76d8d64e5b8aa8e0a2cdf8ee38 [term 1 LEADER]: Becoming Leader. State: Replica: db313c76d8d64e5b8aa8e0a2cdf8ee38, State: Running, Role: LEADER
I20260929 18:41:29.687731 1470 consensus_queue.cc:237] T 997d78333451488ba6063d7c596cf93d P db313c76d8d64e5b8aa8e0a2cdf8ee38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db313c76d8d64e5b8aa8e0a2cdf8ee38" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 35897 } } peers { permanent_uuid: "cfada51e9a7d4c359e1564c522b80ee4" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 35927 } } peers { permanent_uuid: "b987e98b2a834331af07fde1482e62f0" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 40035 } }
I20260929 18:41:29.690961 996 catalog_manager.cc:5495] T 997d78333451488ba6063d7c596cf93d P db313c76d8d64e5b8aa8e0a2cdf8ee38 reported cstate change: term changed from 0 to 1, leader changed from <none> to db313c76d8d64e5b8aa8e0a2cdf8ee38 (127.13.100.3). New cstate: current_term: 1 leader_uuid: "db313c76d8d64e5b8aa8e0a2cdf8ee38" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db313c76d8d64e5b8aa8e0a2cdf8ee38" member_type: VOTER last_known_addr { host: "127.13.100.3" port: 35897 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cfada51e9a7d4c359e1564c522b80ee4" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 35927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b987e98b2a834331af07fde1482e62f0" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 40035 } health_report { overall_health: UNKNOWN } } }
W20260929 18:41:29.708655 1185 tablet.cc:2369] T 997d78333451488ba6063d7c596cf93d P b987e98b2a834331af07fde1482e62f0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:41:29.716413 1458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:29.712953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33544 (local address 127.13.100.3:35897)
0929 18:41:29.713113 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0929 18:41:29.713118 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:41:29.713135 (+ 17us) server_negotiation.cc:407] Connection header received
0929 18:41:29.713210 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:41:29.713213 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:41:29.713266 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:41:29.713379 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:41:29.713615 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:29.714136 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:29.714970 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:29.715180 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:29.715292 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:29.715587 (+ 295us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:41:29.716003 (+ 416us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:41:29.716132 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0929 18:41:29.716196 (+ 64us) server_negotiation.cc:299] Negotiation successful
0929 18:41:29.716247 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 18:41:29.722466 1269 raft_consensus.cc:1261] T 997d78333451488ba6063d7c596cf93d P cfada51e9a7d4c359e1564c522b80ee4 [term 1 FOLLOWER]: Refusing update from remote peer db313c76d8d64e5b8aa8e0a2cdf8ee38: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:41:29.722466 1138 raft_consensus.cc:1261] T 997d78333451488ba6063d7c596cf93d P b987e98b2a834331af07fde1482e62f0 [term 1 FOLLOWER]: Refusing update from remote peer db313c76d8d64e5b8aa8e0a2cdf8ee38: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:41:29.723222 1473 consensus_queue.cc:1035] T 997d78333451488ba6063d7c596cf93d P db313c76d8d64e5b8aa8e0a2cdf8ee38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfada51e9a7d4c359e1564c522b80ee4" member_type: VOTER last_known_addr { host: "127.13.100.2" port: 35927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:41:29.723428 1470 consensus_queue.cc:1035] T 997d78333451488ba6063d7c596cf93d P db313c76d8d64e5b8aa8e0a2cdf8ee38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b987e98b2a834331af07fde1482e62f0" member_type: VOTER last_known_addr { host: "127.13.100.1" port: 40035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:41:29.730604 1476 mvcc.cc:205] Tried to move back new op lower bound from 7334737058695000064 to 7334737058563907584. Current Snapshot: MvccSnapshot[applied={T|T < 7334737058695000064}]
I20260929 18:41:29.732889 1478 mvcc.cc:205] Tried to move back new op lower bound from 7334737058695000064 to 7334737058563907584. Current Snapshot: MvccSnapshot[applied={T|T < 7334737058695000064}]
I20260929 18:41:29.739204 996 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54214:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:41:29.739377 996 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54214:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:41:29.741591 996 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 874a58551e194bbdb820df39e57aac10: Sending DeleteTablet for 3 replicas of tablet 997d78333451488ba6063d7c596cf93d
I20260929 18:41:29.742331 1380 tablet_service.cc:1514] Processing DeleteTablet for tablet 997d78333451488ba6063d7c596cf93d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:41:29 UTC) from {username='slave'} at 127.0.0.1:33534
I20260929 18:41:29.742604 1249 tablet_service.cc:1514] Processing DeleteTablet for tablet 997d78333451488ba6063d7c596cf93d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:41:29 UTC) from {username='slave'} at 127.0.0.1:56894
I20260929 18:41:29.742679 1118 tablet_service.cc:1514] Processing DeleteTablet for tablet 997d78333451488ba6063d7c596cf93d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:41:29 UTC) from {username='slave'} at 127.0.0.1:57372
I20260929 18:41:29.743088 1487 tablet_replica.cc:330] T 997d78333451488ba6063d7c596cf93d P cfada51e9a7d4c359e1564c522b80ee4: stopping tablet replica
I20260929 18:41:29.743166 1488 tablet_replica.cc:330] T 997d78333451488ba6063d7c596cf93d P b987e98b2a834331af07fde1482e62f0: stopping tablet replica
I20260929 18:41:29.743424 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 1056
I20260929 18:41:29.744313 1486 tablet_replica.cc:330] T 997d78333451488ba6063d7c596cf93d P db313c76d8d64e5b8aa8e0a2cdf8ee38: stopping tablet replica
I20260929 18:41:29.744596 1486 raft_consensus.cc:2229] T 997d78333451488ba6063d7c596cf93d P db313c76d8d64e5b8aa8e0a2cdf8ee38 [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:41:29.745010 1486 raft_consensus.cc:2258] T 997d78333451488ba6063d7c596cf93d P db313c76d8d64e5b8aa8e0a2cdf8ee38 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:41:29.748106 1487 raft_consensus.cc:2229] T 997d78333451488ba6063d7c596cf93d P cfada51e9a7d4c359e1564c522b80ee4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:41:29.748315 1486 ts_tablet_manager.cc:1905] T 997d78333451488ba6063d7c596cf93d P db313c76d8d64e5b8aa8e0a2cdf8ee38: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:41:29.748543 1487 raft_consensus.cc:2258] T 997d78333451488ba6063d7c596cf93d P cfada51e9a7d4c359e1564c522b80ee4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:41:29.750124 1487 ts_tablet_manager.cc:1905] T 997d78333451488ba6063d7c596cf93d P cfada51e9a7d4c359e1564c522b80ee4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:41:29.751344 1486 ts_tablet_manager.cc:1918] T 997d78333451488ba6063d7c596cf93d P db313c76d8d64e5b8aa8e0a2cdf8ee38: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:41:29.751472 1486 log.cc:1196] T 997d78333451488ba6063d7c596cf93d P db313c76d8d64e5b8aa8e0a2cdf8ee38: Deleting WAL directory at /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707036041078-13712-0/minicluster-data/ts-2/wal/wals/997d78333451488ba6063d7c596cf93d
I20260929 18:41:29.751816 1486 ts_tablet_manager.cc:1939] T 997d78333451488ba6063d7c596cf93d P db313c76d8d64e5b8aa8e0a2cdf8ee38: Deleting consensus metadata
I20260929 18:41:29.753162 982 catalog_manager.cc:4841] TS db313c76d8d64e5b8aa8e0a2cdf8ee38 (127.13.100.3:35897): tablet 997d78333451488ba6063d7c596cf93d (table test-table [id=dbaaf3fd4f834672b9881db236591ddb]) successfully deleted
I20260929 18:41:29.754117 1487 ts_tablet_manager.cc:1918] T 997d78333451488ba6063d7c596cf93d P cfada51e9a7d4c359e1564c522b80ee4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:41:29.754222 1487 log.cc:1196] T 997d78333451488ba6063d7c596cf93d P cfada51e9a7d4c359e1564c522b80ee4: Deleting WAL directory at /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707036041078-13712-0/minicluster-data/ts-1/wal/wals/997d78333451488ba6063d7c596cf93d
W20260929 18:41:29.754189 982 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.100.1:40035 (error 108)
I20260929 18:41:29.754534 1487 ts_tablet_manager.cc:1939] T 997d78333451488ba6063d7c596cf93d P cfada51e9a7d4c359e1564c522b80ee4: Deleting consensus metadata
I20260929 18:41:29.755364 1461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:29.755039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.1:40035 (local address 127.0.0.1:57386)
0929 18:41:29.755184 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:29.755270 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.1:40035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260929 18:41:29.755386 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 1187
W20260929 18:41:29.756510 982 catalog_manager.cc:4584] TS b987e98b2a834331af07fde1482e62f0 (127.13.100.1:40035): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 997d78333451488ba6063d7c596cf93d: Network error: Client connection negotiation failed: client connection to 127.13.100.1:40035: connect: Connection refused (error 111)
I20260929 18:41:29.762880 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 1318
I20260929 18:41:29.763216 1461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:29.762876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.2:35927 (local address 127.0.0.1:56904)
0929 18:41:29.763038 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:29.763106 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.2:35927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260929 18:41:29.768766 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 965
2026-09-29T18:41:29Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (989 ms)
[----------] 6 tests from EncryptionPolicyTest (10018 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790707036041078-13712-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:41:29 dist-test-slave-j1xs krb5kdc[1492](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:41:29 dist-test-slave-j1xs krb5kdc[1492](info): set up 2 sockets
Sep 29 18:41:29 dist-test-slave-j1xs krb5kdc[1492](info): commencing operation
krb5kdc: starting...
W20260929 18:41:30.830230 13712 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790707036041078-13712-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790707036041078-13712-0/krb5kdc/test-user.keytab.
Sep 29 18:41:30 dist-test-slave-j1xs krb5kdc[1492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707290, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:41:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:41:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790707036041078-13712-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790707036041078-13712-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 18:41:30.948380 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790707036041078-13712-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790707036041078-13712-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790707036041078-13712-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44253
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790707036041078-13712-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44253
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790707036041078-13712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790707036041078-13712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790707036041078-13712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:41:31.062150 1505 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:41:31.062546 1505 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:41:31.062680 1505 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:41:31.068447 1505 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:41:31.068531 1505 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:41:31.068552 1505 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:41:31.068570 1505 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:41:31.068586 1505 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:41:31.074669 1505 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790707036041078-13712-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790707036041078-13712-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44253
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790707036041078-13712-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44253
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1505
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790707036041078-13712-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:41:31.075784 1505 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:41:31.076774 1505 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:41:31.085402 1510 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:41:31.085554 1505 server_base.cc:1034] running on GCE node
W20260929 18:41:31.085392 1513 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:41:31.085392 1511 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:41:31.086284 1505 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:41:31.087392 1505 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:41:31.088557 1505 hybrid_clock.cc:648] HybridClock initialized: now 1790707291088529 us; error 41 us; skew 500 ppm
Sep 29 18:41:31 dist-test-slave-j1xs krb5kdc[1492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707291, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:41:31.091655 1505 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 18:41:31.092581 1505 webserver.cc:460] Webserver started at http://127.0.0.1:35327/ using document root <none> and password file <none>
I20260929 18:41:31.093132 1505 fs_manager.cc:359] Metadata directory not provided
I20260929 18:41:31.093218 1505 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:41:31.093446 1505 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:41:31.095506 1505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790707036041078-13712-0/minicluster-data/master-0/data/instance:
uuid: "34d49666f5814b05aca1ca609d06ed8f"
format_stamp: "Formatted at 2026-09-29 18:41:31 on dist-test-slave-j1xs"
server_key: "07ed586b751a16ce93ce3ec59dd91351"
server_key_iv: "6b5e8860e89c00e6befdd407cda197a4"
server_key_version: "encryptionkey@0"
I20260929 18:41:31.096263 1505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790707036041078-13712-0/minicluster-data/master-0/wal/instance:
uuid: "34d49666f5814b05aca1ca609d06ed8f"
format_stamp: "Formatted at 2026-09-29 18:41:31 on dist-test-slave-j1xs"
server_key: "07ed586b751a16ce93ce3ec59dd91351"
server_key_iv: "6b5e8860e89c00e6befdd407cda197a4"
server_key_version: "encryptionkey@0"
I20260929 18:41:31.100178 1505 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260929 18:41:31.102919 1520 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:41:31.104313 1505 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:41:31.104468 1505 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790707036041078-13712-0/minicluster-data/master-0/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790707036041078-13712-0/minicluster-data/master-0/wal
uuid: "34d49666f5814b05aca1ca609d06ed8f"
format_stamp: "Formatted at 2026-09-29 18:41:31 on dist-test-slave-j1xs"
server_key: "07ed586b751a16ce93ce3ec59dd91351"
server_key_iv: "6b5e8860e89c00e6befdd407cda197a4"
server_key_version: "encryptionkey@0"
I20260929 18:41:31.104586 1505 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790707036041078-13712-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790707036041078-13712-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790707036041078-13712-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:41:31.136521 1505 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:41:31.140542 1505 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:41:31.140769 1505 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:41:31.148196 1505 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44253
I20260929 18:41:31.148214 1572 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44253 every 8 connection(s)
I20260929 18:41:31.149111 1505 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
I20260929 18:41:31.151841 1573 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:41:31.154768 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 1505
I20260929 18:41:31.154918 13712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790707036041078-13712-0/minicluster-data/master-0/wal/instance
I20260929 18:41:31.155220 13712 external_mini_cluster.cc:1428] Setting key 2dc772415f303ce4b9e414efb7f3397b
I20260929 18:41:31.157627 1573 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34d49666f5814b05aca1ca609d06ed8f: Bootstrap starting.
I20260929 18:41:31.160213 1573 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 34d49666f5814b05aca1ca609d06ed8f: Neither blocks nor log segments found. Creating new log.
I20260929 18:41:31.161126 1573 log.cc:824] T 00000000000000000000000000000000 P 34d49666f5814b05aca1ca609d06ed8f: Log is configured to *not* fsync() on all Append() calls
I20260929 18:41:31.163415 1573 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34d49666f5814b05aca1ca609d06ed8f: No bootstrap required, opened a new log
Sep 29 18:41:31 dist-test-slave-j1xs krb5kdc[1492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707290, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 18:41:31.167516 1573 raft_consensus.cc:348] T 00000000000000000000000000000000 P 34d49666f5814b05aca1ca609d06ed8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34d49666f5814b05aca1ca609d06ed8f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44253 } }
I20260929 18:41:31.167800 1573 raft_consensus.cc:374] T 00000000000000000000000000000000 P 34d49666f5814b05aca1ca609d06ed8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:41:31.167873 1573 raft_consensus.cc:729] T 00000000000000000000000000000000 P 34d49666f5814b05aca1ca609d06ed8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34d49666f5814b05aca1ca609d06ed8f, State: Initialized, Role: FOLLOWER
I20260929 18:41:31.168438 1573 consensus_queue.cc:260] T 00000000000000000000000000000000 P 34d49666f5814b05aca1ca609d06ed8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34d49666f5814b05aca1ca609d06ed8f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44253 } }
I20260929 18:41:31.168608 1573 raft_consensus.cc:388] T 00000000000000000000000000000000 P 34d49666f5814b05aca1ca609d06ed8f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:41:31.168679 1573 raft_consensus.cc:482] T 00000000000000000000000000000000 P 34d49666f5814b05aca1ca609d06ed8f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:41:31.168778 1573 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 34d49666f5814b05aca1ca609d06ed8f [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:41:31.169986 1573 raft_consensus.cc:504] T 00000000000000000000000000000000 P 34d49666f5814b05aca1ca609d06ed8f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34d49666f5814b05aca1ca609d06ed8f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44253 } }
I20260929 18:41:31.170341 1573 leader_election.cc:302] T 00000000000000000000000000000000 P 34d49666f5814b05aca1ca609d06ed8f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 34d49666f5814b05aca1ca609d06ed8f; no voters:
I20260929 18:41:31.170655 1573 leader_election.cc:288] T 00000000000000000000000000000000 P 34d49666f5814b05aca1ca609d06ed8f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:41:31.170758 1578 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 34d49666f5814b05aca1ca609d06ed8f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:41:31.171005 1578 raft_consensus.cc:686] T 00000000000000000000000000000000 P 34d49666f5814b05aca1ca609d06ed8f [term 1 LEADER]: Becoming Leader. State: Replica: 34d49666f5814b05aca1ca609d06ed8f, State: Running, Role: LEADER
I20260929 18:41:31.171249 1576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:31.156642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53013 (local address 127.0.0.1:44253)
0929 18:41:31.157031 (+ 389us) server_negotiation.cc:206] Beginning negotiation
0929 18:41:31.157045 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 18:41:31.157079 (+ 34us) server_negotiation.cc:407] Connection header received
0929 18:41:31.157671 (+ 592us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:41:31.157686 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:41:31.158109 (+ 423us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:41:31.158379 (+ 270us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:41:31.159339 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:31.160392 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:31.161562 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:31.162204 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:31.162325 (+ 121us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:31.165127 (+ 2802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:41:31.165135 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:41:31.165151 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:41:31.165184 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:41:31.167805 (+ 2621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:41:31.168329 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:41:31.168334 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:41:31.168339 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:41:31.168483 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:41:31.169003 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:41:31.169009 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:41:31.169014 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:41:31.169511 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:41:31.169726 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0929 18:41:31.170025 (+ 299us) server_negotiation.cc:299] Negotiation successful
0929 18:41:31.170253 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":124,"threads_started":1}
I20260929 18:41:31.171592 1578 consensus_queue.cc:237] T 00000000000000000000000000000000 P 34d49666f5814b05aca1ca609d06ed8f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34d49666f5814b05aca1ca609d06ed8f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44253 } }
I20260929 18:41:31.171897 1573 sys_catalog.cc:564] T 00000000000000000000000000000000 P 34d49666f5814b05aca1ca609d06ed8f [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:41:31.173249 1579 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34d49666f5814b05aca1ca609d06ed8f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 34d49666f5814b05aca1ca609d06ed8f. Latest consensus state: current_term: 1 leader_uuid: "34d49666f5814b05aca1ca609d06ed8f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34d49666f5814b05aca1ca609d06ed8f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44253 } } }
I20260929 18:41:31.173364 1579 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34d49666f5814b05aca1ca609d06ed8f [sys.catalog]: This master's current role is: LEADER
I20260929 18:41:31.173714 1580 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34d49666f5814b05aca1ca609d06ed8f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "34d49666f5814b05aca1ca609d06ed8f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34d49666f5814b05aca1ca609d06ed8f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44253 } } }
I20260929 18:41:31.173813 1580 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34d49666f5814b05aca1ca609d06ed8f [sys.catalog]: This master's current role is: LEADER
I20260929 18:41:31.174549 13712 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260929 18:41:31.177019 1593 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 34d49666f5814b05aca1ca609d06ed8f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:41:31.177093 1593 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:41:31.177152 1587 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:41:31.178123 1587 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:41:31.183986 1587 catalog_manager.cc:1347] Generated new cluster ID: 5847b4e79b1a48638657273eaa8c545c
I20260929 18:41:31.184052 1587 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:41:31.185216 1576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:31.176596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45937 (local address 127.0.0.1:44253)
0929 18:41:31.176701 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0929 18:41:31.176705 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:41:31.176920 (+ 215us) server_negotiation.cc:407] Connection header received
0929 18:41:31.177043 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:41:31.177046 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:41:31.177095 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:41:31.177186 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:41:31.178152 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:31.178971 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:31.180029 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:31.180266 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:31.180374 (+ 108us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:31.181789 (+ 1415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:41:31.181793 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:41:31.181796 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:41:31.181821 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:41:31.183369 (+ 1548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:41:31.183923 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:41:31.183929 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:41:31.183931 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:41:31.184011 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:41:31.184380 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:41:31.184384 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:41:31.184385 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:41:31.184557 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:41:31.184707 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0929 18:41:31.184969 (+ 262us) server_negotiation.cc:299] Negotiation successful
0929 18:41:31.185060 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260929 18:41:31.191404 1587 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:41:31.192250 1587 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:41:31.202932 1587 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 34d49666f5814b05aca1ca609d06ed8f: Generated new TSK 0
I20260929 18:41:31.203601 1587 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:41:31.209149 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 1505
W20260929 18:41:31.214879 1490 connection.cc:535] client connection to 127.0.0.1:44253 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44253): Connection reset by peer (error 104)
2026-09-29T18:41:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1439 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790707036041078-13712-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:41:31 dist-test-slave-j1xs krb5kdc[1604](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:41:31 dist-test-slave-j1xs krb5kdc[1604](info): set up 2 sockets
Sep 29 18:41:31 dist-test-slave-j1xs krb5kdc[1604](info): commencing operation
krb5kdc: starting...
W20260929 18:41:32.250059 13712 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790707036041078-13712-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790707036041078-13712-0/krb5kdc/test-user.keytab.
Sep 29 18:41:32 dist-test-slave-j1xs krb5kdc[1604](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707292, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:41:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:41:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790707036041078-13712-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790707036041078-13712-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 18:41:32.377537 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790707036041078-13712-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790707036041078-13712-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790707036041078-13712-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41155
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790707036041078-13712-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41155
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790707036041078-13712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790707036041078-13712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790707036041078-13712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:41:32.484182 1617 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:41:32.484477 1617 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:41:32.484557 1617 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:41:32.488571 1617 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:41:32.488651 1617 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:41:32.488689 1617 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:41:32.488741 1617 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:41:32.488782 1617 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:41:32.493983 1617 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790707036041078-13712-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790707036041078-13712-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41155
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790707036041078-13712-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41155
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1617
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790707036041078-13712-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:41:32.495029 1617 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:41:32.496013 1617 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:41:32.504489 1623 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:41:32.504635 1617 server_base.cc:1034] running on GCE node
W20260929 18:41:32.504480 1625 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:41:32.504494 1622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:41:32.505242 1617 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:41:32.506426 1617 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:41:32.507612 1617 hybrid_clock.cc:648] HybridClock initialized: now 1790707292507599 us; error 25 us; skew 500 ppm
Sep 29 18:41:32 dist-test-slave-j1xs krb5kdc[1604](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707292, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:41:32.510412 1617 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 18:41:32.511246 1617 webserver.cc:460] Webserver started at http://127.0.0.1:43895/ using document root <none> and password file <none>
I20260929 18:41:32.511782 1617 fs_manager.cc:359] Metadata directory not provided
I20260929 18:41:32.511865 1617 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:41:32.512099 1617 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:41:32.514231 1617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790707036041078-13712-0/minicluster-data/master-0/data/instance:
uuid: "4fe0e147b027461290d3ae1f7bc9f378"
format_stamp: "Formatted at 2026-09-29 18:41:32 on dist-test-slave-j1xs"
server_key: "719704326ed48e37cb9df65cbc881a2c"
server_key_iv: "4aaefc7c5c9e0e6a6ad31b6ce97e18f8"
server_key_version: "encryptionkey@0"
I20260929 18:41:32.514771 1617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790707036041078-13712-0/minicluster-data/master-0/wal/instance:
uuid: "4fe0e147b027461290d3ae1f7bc9f378"
format_stamp: "Formatted at 2026-09-29 18:41:32 on dist-test-slave-j1xs"
server_key: "719704326ed48e37cb9df65cbc881a2c"
server_key_iv: "4aaefc7c5c9e0e6a6ad31b6ce97e18f8"
server_key_version: "encryptionkey@0"
I20260929 18:41:32.518349 1617 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260929 18:41:32.520663 1632 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:41:32.522249 1617 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:41:32.522378 1617 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790707036041078-13712-0/minicluster-data/master-0/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790707036041078-13712-0/minicluster-data/master-0/wal
uuid: "4fe0e147b027461290d3ae1f7bc9f378"
format_stamp: "Formatted at 2026-09-29 18:41:32 on dist-test-slave-j1xs"
server_key: "719704326ed48e37cb9df65cbc881a2c"
server_key_iv: "4aaefc7c5c9e0e6a6ad31b6ce97e18f8"
server_key_version: "encryptionkey@0"
I20260929 18:41:32.522488 1617 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790707036041078-13712-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790707036041078-13712-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790707036041078-13712-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:41:32.540190 1617 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:41:32.543416 1617 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:41:32.543605 1617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:41:32.550103 1617 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41155
I20260929 18:41:32.550130 1684 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41155 every 8 connection(s)
I20260929 18:41:32.550977 1617 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
I20260929 18:41:32.553398 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 1617
I20260929 18:41:32.553500 13712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790707036041078-13712-0/minicluster-data/master-0/wal/instance
I20260929 18:41:32.553561 1685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:41:32.553810 13712 external_mini_cluster.cc:1428] Setting key 5bbd2e1844fea41de1b7dc7696a23006
I20260929 18:41:32.560468 1685 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fe0e147b027461290d3ae1f7bc9f378: Bootstrap starting.
I20260929 18:41:32.563091 1685 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4fe0e147b027461290d3ae1f7bc9f378: Neither blocks nor log segments found. Creating new log.
I20260929 18:41:32.563318 1688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:32.555126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46015 (local address 127.0.0.1:41155)
0929 18:41:32.555545 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0929 18:41:32.555566 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0929 18:41:32.555605 (+ 39us) server_negotiation.cc:407] Connection header received
0929 18:41:32.556324 (+ 719us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:41:32.556345 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:41:32.556765 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:41:32.557114 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:41:32.557516 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:32.558973 (+ 1457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:32.559784 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:32.560431 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:32.560615 (+ 184us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:32.561437 (+ 822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:41:32.561443 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:41:32.561466 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:41:32.561980 (+ 514us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:41:32.562043 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:41:32.562070 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:41:32.562195 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0929 18:41:32.562316 (+ 121us) server_negotiation.cc:299] Negotiation successful
0929 18:41:32.562432 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":117,"threads_started":1}
I20260929 18:41:32.563856 1685 log.cc:824] T 00000000000000000000000000000000 P 4fe0e147b027461290d3ae1f7bc9f378: Log is configured to *not* fsync() on all Append() calls
I20260929 18:41:32.565855 1685 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fe0e147b027461290d3ae1f7bc9f378: No bootstrap required, opened a new log
I20260929 18:41:32.566015 13712 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:41:32.570065 1685 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4fe0e147b027461290d3ae1f7bc9f378 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fe0e147b027461290d3ae1f7bc9f378" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41155 } }
I20260929 18:41:32.570290 1685 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4fe0e147b027461290d3ae1f7bc9f378 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:41:32.570348 1685 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4fe0e147b027461290d3ae1f7bc9f378 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fe0e147b027461290d3ae1f7bc9f378, State: Initialized, Role: FOLLOWER
I20260929 18:41:32.570746 1685 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4fe0e147b027461290d3ae1f7bc9f378 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fe0e147b027461290d3ae1f7bc9f378" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41155 } }
I20260929 18:41:32.570889 1685 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4fe0e147b027461290d3ae1f7bc9f378 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:41:32.570946 1685 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4fe0e147b027461290d3ae1f7bc9f378 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:41:32.571048 1685 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4fe0e147b027461290d3ae1f7bc9f378 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:41:32.572165 1685 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4fe0e147b027461290d3ae1f7bc9f378 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fe0e147b027461290d3ae1f7bc9f378" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41155 } }
I20260929 18:41:32.572501 1688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:32.567870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56355 (local address 127.0.0.1:41155)
0929 18:41:32.567982 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0929 18:41:32.567987 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:41:32.568670 (+ 683us) server_negotiation.cc:407] Connection header received
0929 18:41:32.568742 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:41:32.568747 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:41:32.568789 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:41:32.568872 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:41:32.569089 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:32.569828 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:32.570753 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:32.571099 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:32.571204 (+ 105us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:32.572110 (+ 906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:41:32.572113 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:41:32.572115 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:41:32.572125 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:41:32.572146 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:41:32.572156 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:41:32.572224 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 18:41:32.572383 (+ 159us) server_negotiation.cc:299] Negotiation successful
0929 18:41:32.572410 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 18:41:32.572532 1685 leader_election.cc:302] T 00000000000000000000000000000000 P 4fe0e147b027461290d3ae1f7bc9f378 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4fe0e147b027461290d3ae1f7bc9f378; no voters:
I20260929 18:41:32.573132 1685 leader_election.cc:288] T 00000000000000000000000000000000 P 4fe0e147b027461290d3ae1f7bc9f378 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:41:32.573217 1695 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4fe0e147b027461290d3ae1f7bc9f378 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:41:32.573428 1695 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4fe0e147b027461290d3ae1f7bc9f378 [term 1 LEADER]: Becoming Leader. State: Replica: 4fe0e147b027461290d3ae1f7bc9f378, State: Running, Role: LEADER
I20260929 18:41:32.573711 1695 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4fe0e147b027461290d3ae1f7bc9f378 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fe0e147b027461290d3ae1f7bc9f378" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41155 } }
I20260929 18:41:32.574421 1685 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4fe0e147b027461290d3ae1f7bc9f378 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:41:32.574800 1697 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fe0e147b027461290d3ae1f7bc9f378 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4fe0e147b027461290d3ae1f7bc9f378" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fe0e147b027461290d3ae1f7bc9f378" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41155 } } }
I20260929 18:41:32.574914 1697 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fe0e147b027461290d3ae1f7bc9f378 [sys.catalog]: This master's current role is: LEADER
I20260929 18:41:32.575134 1696 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fe0e147b027461290d3ae1f7bc9f378 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4fe0e147b027461290d3ae1f7bc9f378. Latest consensus state: current_term: 1 leader_uuid: "4fe0e147b027461290d3ae1f7bc9f378" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fe0e147b027461290d3ae1f7bc9f378" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41155 } } }
I20260929 18:41:32.575232 1696 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fe0e147b027461290d3ae1f7bc9f378 [sys.catalog]: This master's current role is: LEADER
I20260929 18:41:32.575341 1699 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:41:32.578267 1699 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:41:32.584033 1699 catalog_manager.cc:1347] Generated new cluster ID: cd5220ba8fe44da7984fb89e195ca6b6
I20260929 18:41:32.584090 1699 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:41:32.598249 1699 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:41:32.599335 1699 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:41:32.604957 1699 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4fe0e147b027461290d3ae1f7bc9f378: Generated new TSK 0
I20260929 18:41:32.605571 1699 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:41:32.618769 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 1617
W20260929 18:41:32.624050 1602 connection.cc:535] client connection to 127.0.0.1:41155 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41155): Connection reset by peer (error 104)
2026-09-29T18:41:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1408 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790707036041078-13712-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:41:32 dist-test-slave-j1xs krb5kdc[1716](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:41:32 dist-test-slave-j1xs krb5kdc[1716](info): set up 2 sockets
Sep 29 18:41:32 dist-test-slave-j1xs krb5kdc[1716](info): commencing operation
krb5kdc: starting...
W20260929 18:41:33.654175 13712 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790707036041078-13712-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790707036041078-13712-0/krb5kdc/test-user.keytab.
Sep 29 18:41:33 dist-test-slave-j1xs krb5kdc[1716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707293, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:41:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:41:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790707036041078-13712-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790707036041078-13712-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 18:41:33.773692 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790707036041078-13712-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790707036041078-13712-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790707036041078-13712-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42879
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790707036041078-13712-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42879
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790707036041078-13712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790707036041078-13712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790707036041078-13712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:41:33.884644 1729 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:41:33.885020 1729 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:41:33.885113 1729 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:41:33.889196 1729 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:41:33.889254 1729 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:41:33.889274 1729 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:41:33.889292 1729 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:41:33.889346 1729 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:41:33.894683 1729 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790707036041078-13712-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790707036041078-13712-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42879
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790707036041078-13712-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42879
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1729
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790707036041078-13712-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:41:33.895761 1729 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:41:33.896948 1729 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:41:33.905431 1737 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:41:33.905542 1729 server_base.cc:1034] running on GCE node
W20260929 18:41:33.905431 1734 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:41:33.905443 1735 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:41:33.906303 1729 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:41:33.907392 1729 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:41:33.908591 1729 hybrid_clock.cc:648] HybridClock initialized: now 1790707293908574 us; error 27 us; skew 500 ppm
Sep 29 18:41:33 dist-test-slave-j1xs krb5kdc[1716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707293, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:41:33.911527 1729 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 18:41:33.912451 1729 webserver.cc:460] Webserver started at http://127.0.0.1:36257/ using document root <none> and password file <none>
I20260929 18:41:33.913071 1729 fs_manager.cc:359] Metadata directory not provided
I20260929 18:41:33.913156 1729 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:41:33.913357 1729 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:41:33.915407 1729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790707036041078-13712-0/minicluster-data/master-0/data/instance:
uuid: "50b59be7b9ce41618c6131e5fa27513d"
format_stamp: "Formatted at 2026-09-29 18:41:33 on dist-test-slave-j1xs"
server_key: "41b1b1a4adb28c13de4564263cf382c7"
server_key_iv: "2dcb6540b1f72edc8e56f003958f2c24"
server_key_version: "encryptionkey@0"
I20260929 18:41:33.915964 1729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790707036041078-13712-0/minicluster-data/master-0/wal/instance:
uuid: "50b59be7b9ce41618c6131e5fa27513d"
format_stamp: "Formatted at 2026-09-29 18:41:33 on dist-test-slave-j1xs"
server_key: "41b1b1a4adb28c13de4564263cf382c7"
server_key_iv: "2dcb6540b1f72edc8e56f003958f2c24"
server_key_version: "encryptionkey@0"
I20260929 18:41:33.919677 1729 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 18:41:33.922107 1744 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:41:33.923357 1729 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:41:33.923499 1729 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790707036041078-13712-0/minicluster-data/master-0/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790707036041078-13712-0/minicluster-data/master-0/wal
uuid: "50b59be7b9ce41618c6131e5fa27513d"
format_stamp: "Formatted at 2026-09-29 18:41:33 on dist-test-slave-j1xs"
server_key: "41b1b1a4adb28c13de4564263cf382c7"
server_key_iv: "2dcb6540b1f72edc8e56f003958f2c24"
server_key_version: "encryptionkey@0"
I20260929 18:41:33.923609 1729 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790707036041078-13712-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790707036041078-13712-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790707036041078-13712-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:41:33.928253 1729 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:41:33.931489 1729 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:41:33.931680 1729 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:41:33.938581 1729 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42879
I20260929 18:41:33.938611 1796 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42879 every 8 connection(s)
I20260929 18:41:33.939482 1729 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
I20260929 18:41:33.939709 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 1729
I20260929 18:41:33.939836 13712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790707036041078-13712-0/minicluster-data/master-0/wal/instance
I20260929 18:41:33.940191 13712 external_mini_cluster.cc:1428] Setting key 6b9b9b8e8798a639f46f4e0c16d9a8ed
I20260929 18:41:33.942698 1797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:41:33.945080 1800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:33.941501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50671 (local address 127.0.0.1:42879)
0929 18:41:33.941958 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0929 18:41:33.941973 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 18:41:33.942006 (+ 33us) server_negotiation.cc:407] Connection header received
0929 18:41:33.942630 (+ 624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:41:33.942645 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:41:33.942954 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:41:33.943277 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:41:33.944002 (+ 725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:41:33.944007 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:41:33.944018 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:41:33.944028 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:41:33.944076 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:41:33.944093 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:41:33.944233 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0929 18:41:33.944301 (+ 68us) server_negotiation.cc:299] Negotiation successful
0929 18:41:33.944443 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":138,"threads_started":1}
I20260929 18:41:33.947870 13712 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:41:33.948800 1797 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50b59be7b9ce41618c6131e5fa27513d: Bootstrap starting.
I20260929 18:41:33.951406 1797 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 50b59be7b9ce41618c6131e5fa27513d: Neither blocks nor log segments found. Creating new log.
I20260929 18:41:33.951589 1800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:33.949795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33171 (local address 127.0.0.1:42879)
0929 18:41:33.949933 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0929 18:41:33.949939 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:41:33.950139 (+ 200us) server_negotiation.cc:407] Connection header received
0929 18:41:33.950276 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:41:33.950280 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:41:33.950316 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:41:33.950392 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:41:33.951152 (+ 760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:41:33.951155 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:41:33.951157 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:41:33.951168 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:41:33.951194 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:41:33.951210 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:41:33.951292 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 18:41:33.951365 (+ 73us) server_negotiation.cc:299] Negotiation successful
0929 18:41:33.951401 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260929 18:41:33.952322 1797 log.cc:824] T 00000000000000000000000000000000 P 50b59be7b9ce41618c6131e5fa27513d: Log is configured to *not* fsync() on all Append() calls
I20260929 18:41:33.954432 1797 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50b59be7b9ce41618c6131e5fa27513d: No bootstrap required, opened a new log
I20260929 18:41:33.957808 1797 raft_consensus.cc:348] T 00000000000000000000000000000000 P 50b59be7b9ce41618c6131e5fa27513d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50b59be7b9ce41618c6131e5fa27513d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42879 } }
I20260929 18:41:33.958098 1797 raft_consensus.cc:374] T 00000000000000000000000000000000 P 50b59be7b9ce41618c6131e5fa27513d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:41:33.958146 1797 raft_consensus.cc:729] T 00000000000000000000000000000000 P 50b59be7b9ce41618c6131e5fa27513d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50b59be7b9ce41618c6131e5fa27513d, State: Initialized, Role: FOLLOWER
I20260929 18:41:33.958530 1797 consensus_queue.cc:260] T 00000000000000000000000000000000 P 50b59be7b9ce41618c6131e5fa27513d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50b59be7b9ce41618c6131e5fa27513d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42879 } }
I20260929 18:41:33.958639 1797 raft_consensus.cc:388] T 00000000000000000000000000000000 P 50b59be7b9ce41618c6131e5fa27513d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:41:33.958678 1797 raft_consensus.cc:482] T 00000000000000000000000000000000 P 50b59be7b9ce41618c6131e5fa27513d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:41:33.958740 1797 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 50b59be7b9ce41618c6131e5fa27513d [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:41:33.959650 1797 raft_consensus.cc:504] T 00000000000000000000000000000000 P 50b59be7b9ce41618c6131e5fa27513d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50b59be7b9ce41618c6131e5fa27513d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42879 } }
I20260929 18:41:33.959942 1797 leader_election.cc:302] T 00000000000000000000000000000000 P 50b59be7b9ce41618c6131e5fa27513d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 50b59be7b9ce41618c6131e5fa27513d; no voters:
I20260929 18:41:33.960207 1797 leader_election.cc:288] T 00000000000000000000000000000000 P 50b59be7b9ce41618c6131e5fa27513d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:41:33.960393 1807 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 50b59be7b9ce41618c6131e5fa27513d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:41:33.960676 1807 raft_consensus.cc:686] T 00000000000000000000000000000000 P 50b59be7b9ce41618c6131e5fa27513d [term 1 LEADER]: Becoming Leader. State: Replica: 50b59be7b9ce41618c6131e5fa27513d, State: Running, Role: LEADER
I20260929 18:41:33.960991 1807 consensus_queue.cc:237] T 00000000000000000000000000000000 P 50b59be7b9ce41618c6131e5fa27513d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50b59be7b9ce41618c6131e5fa27513d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42879 } }
I20260929 18:41:33.961542 1797 sys_catalog.cc:564] T 00000000000000000000000000000000 P 50b59be7b9ce41618c6131e5fa27513d [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:41:33.962901 1809 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50b59be7b9ce41618c6131e5fa27513d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 50b59be7b9ce41618c6131e5fa27513d. Latest consensus state: current_term: 1 leader_uuid: "50b59be7b9ce41618c6131e5fa27513d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50b59be7b9ce41618c6131e5fa27513d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42879 } } }
I20260929 18:41:33.962901 1808 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50b59be7b9ce41618c6131e5fa27513d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "50b59be7b9ce41618c6131e5fa27513d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50b59be7b9ce41618c6131e5fa27513d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42879 } } }
I20260929 18:41:33.963085 1808 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50b59be7b9ce41618c6131e5fa27513d [sys.catalog]: This master's current role is: LEADER
I20260929 18:41:33.963371 1809 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50b59be7b9ce41618c6131e5fa27513d [sys.catalog]: This master's current role is: LEADER
I20260929 18:41:33.963479 1816 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:41:33.966253 1816 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:41:33.971976 1816 catalog_manager.cc:1347] Generated new cluster ID: dcc94ccaf66f42bc994d2c8e7508852f
I20260929 18:41:33.972051 1816 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:41:33.982329 1816 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:41:33.982787 1816 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:41:33.989840 1816 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 50b59be7b9ce41618c6131e5fa27513d: Generated new TSK 0
I20260929 18:41:33.990527 1816 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:41:33.993479 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 1729
2026-09-29T18:41:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1374 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790707036041078-13712-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:41:34 dist-test-slave-j1xs krb5kdc[1828](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:41:34 dist-test-slave-j1xs krb5kdc[1828](info): set up 2 sockets
Sep 29 18:41:34 dist-test-slave-j1xs krb5kdc[1828](info): commencing operation
krb5kdc: starting...
W20260929 18:41:35.026108 13712 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790707036041078-13712-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790707036041078-13712-0/krb5kdc/test-user.keytab.
Sep 29 18:41:35 dist-test-slave-j1xs krb5kdc[1828](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707295, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:41:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:41:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790707036041078-13712-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790707036041078-13712-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 18:41:35.150699 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790707036041078-13712-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790707036041078-13712-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790707036041078-13712-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33187
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790707036041078-13712-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33187
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790707036041078-13712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790707036041078-13712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790707036041078-13712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:41:35.264288 1841 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:41:35.264637 1841 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:41:35.264683 1841 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:41:35.268926 1841 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:41:35.269001 1841 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:41:35.269021 1841 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:41:35.269038 1841 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:41:35.269059 1841 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:41:35.274581 1841 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790707036041078-13712-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790707036041078-13712-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33187
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790707036041078-13712-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33187
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1841
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790707036041078-13712-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:41:35.275645 1841 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:41:35.276610 1841 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:41:35.285197 1846 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:41:35.285192 1849 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:41:35.285192 1847 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:41:35.285444 1841 server_base.cc:1034] running on GCE node
I20260929 18:41:35.286068 1841 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:41:35.287199 1841 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:41:35.288372 1841 hybrid_clock.cc:648] HybridClock initialized: now 1790707295288353 us; error 27 us; skew 500 ppm
Sep 29 18:41:35 dist-test-slave-j1xs krb5kdc[1828](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707295, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:41:35.296288 1841 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 18:41:35.297521 1841 webserver.cc:460] Webserver started at http://127.0.0.1:42313/ using document root <none> and password file <none>
I20260929 18:41:35.298225 1841 fs_manager.cc:359] Metadata directory not provided
I20260929 18:41:35.298293 1841 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:41:35.298503 1841 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:41:35.300547 1841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790707036041078-13712-0/minicluster-data/master-0/data/instance:
uuid: "eda7b7f2378e43cbae1e59d3ab7bae46"
format_stamp: "Formatted at 2026-09-29 18:41:35 on dist-test-slave-j1xs"
server_key: "f8875b13990dd306e3889884e43c9eeb"
server_key_iv: "77887b68da4a856d6f3dd5c5d3a99089"
server_key_version: "encryptionkey@0"
I20260929 18:41:35.301146 1841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790707036041078-13712-0/minicluster-data/master-0/wal/instance:
uuid: "eda7b7f2378e43cbae1e59d3ab7bae46"
format_stamp: "Formatted at 2026-09-29 18:41:35 on dist-test-slave-j1xs"
server_key: "f8875b13990dd306e3889884e43c9eeb"
server_key_iv: "77887b68da4a856d6f3dd5c5d3a99089"
server_key_version: "encryptionkey@0"
I20260929 18:41:35.305047 1841 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 18:41:35.307704 1856 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:41:35.308985 1841 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:41:35.309151 1841 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790707036041078-13712-0/minicluster-data/master-0/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790707036041078-13712-0/minicluster-data/master-0/wal
uuid: "eda7b7f2378e43cbae1e59d3ab7bae46"
format_stamp: "Formatted at 2026-09-29 18:41:35 on dist-test-slave-j1xs"
server_key: "f8875b13990dd306e3889884e43c9eeb"
server_key_iv: "77887b68da4a856d6f3dd5c5d3a99089"
server_key_version: "encryptionkey@0"
I20260929 18:41:35.309252 1841 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790707036041078-13712-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790707036041078-13712-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790707036041078-13712-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:41:35.335731 1841 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:41:35.339129 1841 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:41:35.339339 1841 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:41:35.346787 1841 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33187
I20260929 18:41:35.346867 1908 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33187 every 8 connection(s)
I20260929 18:41:35.347743 1841 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
I20260929 18:41:35.350528 1909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:41:35.356326 1909 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eda7b7f2378e43cbae1e59d3ab7bae46: Bootstrap starting.
I20260929 18:41:35.357616 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 1841
I20260929 18:41:35.357762 13712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790707036041078-13712-0/minicluster-data/master-0/wal/instance
I20260929 18:41:35.358081 13712 external_mini_cluster.cc:1428] Setting key d2ad7139b327f92cc9a2b2aece16b4c1
I20260929 18:41:35.359082 1909 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eda7b7f2378e43cbae1e59d3ab7bae46: Neither blocks nor log segments found. Creating new log.
I20260929 18:41:35.359881 1909 log.cc:824] T 00000000000000000000000000000000 P eda7b7f2378e43cbae1e59d3ab7bae46: Log is configured to *not* fsync() on all Append() calls
I20260929 18:41:35.362174 1909 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eda7b7f2378e43cbae1e59d3ab7bae46: No bootstrap required, opened a new log
I20260929 18:41:35.366331 1909 raft_consensus.cc:348] T 00000000000000000000000000000000 P eda7b7f2378e43cbae1e59d3ab7bae46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eda7b7f2378e43cbae1e59d3ab7bae46" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33187 } }
I20260929 18:41:35.366641 1909 raft_consensus.cc:374] T 00000000000000000000000000000000 P eda7b7f2378e43cbae1e59d3ab7bae46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:41:35.366709 1909 raft_consensus.cc:729] T 00000000000000000000000000000000 P eda7b7f2378e43cbae1e59d3ab7bae46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eda7b7f2378e43cbae1e59d3ab7bae46, State: Initialized, Role: FOLLOWER
I20260929 18:41:35.367228 1909 consensus_queue.cc:260] T 00000000000000000000000000000000 P eda7b7f2378e43cbae1e59d3ab7bae46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eda7b7f2378e43cbae1e59d3ab7bae46" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33187 } }
I20260929 18:41:35.367393 1909 raft_consensus.cc:388] T 00000000000000000000000000000000 P eda7b7f2378e43cbae1e59d3ab7bae46 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
Sep 29 18:41:35 dist-test-slave-j1xs krb5kdc[1828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707295, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 18:41:35.367475 1909 raft_consensus.cc:482] T 00000000000000000000000000000000 P eda7b7f2378e43cbae1e59d3ab7bae46 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:41:35.367587 1909 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eda7b7f2378e43cbae1e59d3ab7bae46 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:41:35.368530 1909 raft_consensus.cc:504] T 00000000000000000000000000000000 P eda7b7f2378e43cbae1e59d3ab7bae46 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eda7b7f2378e43cbae1e59d3ab7bae46" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33187 } }
I20260929 18:41:35.368870 1909 leader_election.cc:302] T 00000000000000000000000000000000 P eda7b7f2378e43cbae1e59d3ab7bae46 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eda7b7f2378e43cbae1e59d3ab7bae46; no voters:
I20260929 18:41:35.369233 1909 leader_election.cc:288] T 00000000000000000000000000000000 P eda7b7f2378e43cbae1e59d3ab7bae46 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:41:35.369371 1914 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eda7b7f2378e43cbae1e59d3ab7bae46 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:41:35.369607 1914 raft_consensus.cc:686] T 00000000000000000000000000000000 P eda7b7f2378e43cbae1e59d3ab7bae46 [term 1 LEADER]: Becoming Leader. State: Replica: eda7b7f2378e43cbae1e59d3ab7bae46, State: Running, Role: LEADER
I20260929 18:41:35.369958 1914 consensus_queue.cc:237] T 00000000000000000000000000000000 P eda7b7f2378e43cbae1e59d3ab7bae46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eda7b7f2378e43cbae1e59d3ab7bae46" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33187 } }
I20260929 18:41:35.370522 1909 sys_catalog.cc:564] T 00000000000000000000000000000000 P eda7b7f2378e43cbae1e59d3ab7bae46 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:41:35.372092 1916 sys_catalog.cc:455] T 00000000000000000000000000000000 P eda7b7f2378e43cbae1e59d3ab7bae46 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eda7b7f2378e43cbae1e59d3ab7bae46. Latest consensus state: current_term: 1 leader_uuid: "eda7b7f2378e43cbae1e59d3ab7bae46" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eda7b7f2378e43cbae1e59d3ab7bae46" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33187 } } }
I20260929 18:41:35.372207 1916 sys_catalog.cc:458] T 00000000000000000000000000000000 P eda7b7f2378e43cbae1e59d3ab7bae46 [sys.catalog]: This master's current role is: LEADER
I20260929 18:41:35.372512 1915 sys_catalog.cc:455] T 00000000000000000000000000000000 P eda7b7f2378e43cbae1e59d3ab7bae46 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eda7b7f2378e43cbae1e59d3ab7bae46" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eda7b7f2378e43cbae1e59d3ab7bae46" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33187 } } }
I20260929 18:41:35.373457 1915 sys_catalog.cc:458] T 00000000000000000000000000000000 P eda7b7f2378e43cbae1e59d3ab7bae46 [sys.catalog]: This master's current role is: LEADER
I20260929 18:41:35.373728 1928 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:41:35.374250 1912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:35.359217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41437 (local address 127.0.0.1:33187)
0929 18:41:35.359650 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0929 18:41:35.359663 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 18:41:35.359696 (+ 33us) server_negotiation.cc:407] Connection header received
0929 18:41:35.360602 (+ 906us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:41:35.360627 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:41:35.360974 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:41:35.361303 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:41:35.362308 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:35.363284 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:35.364463 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:35.364962 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:35.365099 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:35.368112 (+ 3013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:41:35.368118 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:41:35.368130 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:41:35.368157 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:41:35.370354 (+ 2197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:41:35.370840 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:41:35.370847 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:41:35.370852 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:41:35.371001 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:41:35.371360 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:41:35.371364 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:41:35.371366 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:41:35.371900 (+ 534us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:41:35.372126 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0929 18:41:35.373067 (+ 941us) server_negotiation.cc:299] Negotiation successful
0929 18:41:35.373333 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":130,"threads_started":1}
I20260929 18:41:35.376760 1928 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:41:35.376967 13712 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:41:35.382553 1928 catalog_manager.cc:1347] Generated new cluster ID: d43f2066a0b342ab841836ec1dbbf033
I20260929 18:41:35.382620 1928 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:41:35.391644 1912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:35.382804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48125 (local address 127.0.0.1:33187)
0929 18:41:35.382912 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0929 18:41:35.382916 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:41:35.383299 (+ 383us) server_negotiation.cc:407] Connection header received
0929 18:41:35.383488 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:41:35.383492 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:41:35.383536 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:41:35.383610 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:41:35.384428 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:35.385224 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:35.386394 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:35.386723 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:35.386814 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:35.388390 (+ 1576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:41:35.388393 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:41:35.388396 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:41:35.388418 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:41:35.390054 (+ 1636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:41:35.390491 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:41:35.390494 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:41:35.390496 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:41:35.390568 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:41:35.390847 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:41:35.390850 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:41:35.390851 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:41:35.391035 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:41:35.391146 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 18:41:35.391376 (+ 230us) server_negotiation.cc:299] Negotiation successful
0929 18:41:35.391482 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260929 18:41:35.394058 1928 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:41:35.395035 1928 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:41:35.400079 1928 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eda7b7f2378e43cbae1e59d3ab7bae46: Generated new TSK 0
I20260929 18:41:35.400753 1928 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:41:35.404495 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 1841
2026-09-29T18:41:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1411 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790707036041078-13712-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:41:35 dist-test-slave-j1xs krb5kdc[1940](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:41:35 dist-test-slave-j1xs krb5kdc[1940](info): set up 2 sockets
Sep 29 18:41:35 dist-test-slave-j1xs krb5kdc[1940](info): commencing operation
krb5kdc: starting...
W20260929 18:41:36.450228 13712 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790707036041078-13712-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790707036041078-13712-0/krb5kdc/test-user.keytab.
Sep 29 18:41:36 dist-test-slave-j1xs krb5kdc[1940](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707296, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:41:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:41:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790707036041078-13712-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790707036041078-13712-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 18:41:36.568251 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790707036041078-13712-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790707036041078-13712-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790707036041078-13712-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32891
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790707036041078-13712-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32891
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790707036041078-13712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790707036041078-13712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790707036041078-13712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:41:36.678834 1953 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:41:36.679086 1953 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:41:36.679126 1953 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:41:36.683117 1953 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:41:36.683172 1953 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:41:36.683193 1953 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:41:36.683240 1953 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:41:36.683277 1953 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:41:36.688474 1953 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790707036041078-13712-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790707036041078-13712-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32891
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790707036041078-13712-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32891
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1953
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790707036041078-13712-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:41:36.689472 1953 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:41:36.690501 1953 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:41:36.699410 1961 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:41:36.699424 1959 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:41:36.699421 1958 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:41:36.699640 1953 server_base.cc:1034] running on GCE node
I20260929 18:41:36.700271 1953 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:41:36.701411 1953 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:41:36.702574 1953 hybrid_clock.cc:648] HybridClock initialized: now 1790707296702561 us; error 26 us; skew 500 ppm
Sep 29 18:41:36 dist-test-slave-j1xs krb5kdc[1940](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707296, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:41:36.705476 1953 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 18:41:36.706390 1953 webserver.cc:460] Webserver started at http://127.0.0.1:37027/ using document root <none> and password file <none>
I20260929 18:41:36.706938 1953 fs_manager.cc:359] Metadata directory not provided
I20260929 18:41:36.707000 1953 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:41:36.707202 1953 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:41:36.709179 1953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790707036041078-13712-0/minicluster-data/master-0/data/instance:
uuid: "9347aa43aa114de5bcdd634f0b35f252"
format_stamp: "Formatted at 2026-09-29 18:41:36 on dist-test-slave-j1xs"
server_key: "d4e142e13f2e4446f454469ff7ecbbfa"
server_key_iv: "4faeaff3df75e4addec8857593fb5e30"
server_key_version: "encryptionkey@0"
I20260929 18:41:36.709730 1953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790707036041078-13712-0/minicluster-data/master-0/wal/instance:
uuid: "9347aa43aa114de5bcdd634f0b35f252"
format_stamp: "Formatted at 2026-09-29 18:41:36 on dist-test-slave-j1xs"
server_key: "d4e142e13f2e4446f454469ff7ecbbfa"
server_key_iv: "4faeaff3df75e4addec8857593fb5e30"
server_key_version: "encryptionkey@0"
I20260929 18:41:36.713459 1953 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260929 18:41:36.716014 1968 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:41:36.717335 1953 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:41:36.717499 1953 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790707036041078-13712-0/minicluster-data/master-0/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790707036041078-13712-0/minicluster-data/master-0/wal
uuid: "9347aa43aa114de5bcdd634f0b35f252"
format_stamp: "Formatted at 2026-09-29 18:41:36 on dist-test-slave-j1xs"
server_key: "d4e142e13f2e4446f454469ff7ecbbfa"
server_key_iv: "4faeaff3df75e4addec8857593fb5e30"
server_key_version: "encryptionkey@0"
I20260929 18:41:36.717617 1953 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790707036041078-13712-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790707036041078-13712-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790707036041078-13712-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:41:36.732555 1953 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:41:36.735666 1953 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:41:36.735884 1953 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:41:36.742925 1953 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32891
I20260929 18:41:36.742954 2020 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32891 every 8 connection(s)
I20260929 18:41:36.743867 1953 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
I20260929 18:41:36.744485 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 1953
I20260929 18:41:36.744629 13712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790707036041078-13712-0/minicluster-data/master-0/wal/instance
I20260929 18:41:36.745012 13712 external_mini_cluster.cc:1428] Setting key fecb68cb15046e6cde7e6cb5ddc691d0
I20260929 18:41:36.747174 2021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:41:36.753228 2021 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9347aa43aa114de5bcdd634f0b35f252: Bootstrap starting.
I20260929 18:41:36.754029 2024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:36.746441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56645 (local address 127.0.0.1:32891)
0929 18:41:36.746866 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0929 18:41:36.746883 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 18:41:36.746919 (+ 36us) server_negotiation.cc:407] Connection header received
0929 18:41:36.747601 (+ 682us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:41:36.747618 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:41:36.747950 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:41:36.748223 (+ 273us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:41:36.748582 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:36.749554 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:36.750774 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:36.751277 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:36.751401 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:36.752293 (+ 892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:41:36.752298 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:41:36.752309 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:41:36.752688 (+ 379us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:41:36.752740 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:41:36.752762 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:41:36.752845 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 18:41:36.753015 (+ 170us) server_negotiation.cc:299] Negotiation successful
0929 18:41:36.753247 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":148,"threads_started":1}
I20260929 18:41:36.755527 2021 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9347aa43aa114de5bcdd634f0b35f252: Neither blocks nor log segments found. Creating new log.
I20260929 18:41:36.756417 2021 log.cc:824] T 00000000000000000000000000000000 P 9347aa43aa114de5bcdd634f0b35f252: Log is configured to *not* fsync() on all Append() calls
I20260929 18:41:36.756831 13712 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:41:36.758687 2021 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9347aa43aa114de5bcdd634f0b35f252: No bootstrap required, opened a new log
I20260929 18:41:36.762897 2024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:36.758885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48881 (local address 127.0.0.1:32891)
0929 18:41:36.759015 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0929 18:41:36.759022 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 18:41:36.759284 (+ 262us) server_negotiation.cc:407] Connection header received
0929 18:41:36.759398 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:41:36.759400 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:41:36.759435 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:41:36.759501 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:41:36.759763 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:36.760244 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:36.761443 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:36.761655 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:36.761743 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:36.762504 (+ 761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:41:36.762507 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:41:36.762509 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:41:36.762519 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:41:36.762541 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:41:36.762550 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:41:36.762611 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0929 18:41:36.762718 (+ 107us) server_negotiation.cc:299] Negotiation successful
0929 18:41:36.762757 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 18:41:36.763077 2021 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9347aa43aa114de5bcdd634f0b35f252 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9347aa43aa114de5bcdd634f0b35f252" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32891 } }
I20260929 18:41:36.763335 2021 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9347aa43aa114de5bcdd634f0b35f252 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:41:36.763404 2021 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9347aa43aa114de5bcdd634f0b35f252 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9347aa43aa114de5bcdd634f0b35f252, State: Initialized, Role: FOLLOWER
I20260929 18:41:36.763826 2021 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9347aa43aa114de5bcdd634f0b35f252 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9347aa43aa114de5bcdd634f0b35f252" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32891 } }
I20260929 18:41:36.763952 2021 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9347aa43aa114de5bcdd634f0b35f252 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:41:36.763993 2021 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9347aa43aa114de5bcdd634f0b35f252 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:41:36.764060 2021 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9347aa43aa114de5bcdd634f0b35f252 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:41:36.764815 2021 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9347aa43aa114de5bcdd634f0b35f252 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9347aa43aa114de5bcdd634f0b35f252" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32891 } }
I20260929 18:41:36.765098 2021 leader_election.cc:302] T 00000000000000000000000000000000 P 9347aa43aa114de5bcdd634f0b35f252 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9347aa43aa114de5bcdd634f0b35f252; no voters:
I20260929 18:41:36.765326 2021 leader_election.cc:288] T 00000000000000000000000000000000 P 9347aa43aa114de5bcdd634f0b35f252 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:41:36.765439 2031 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9347aa43aa114de5bcdd634f0b35f252 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:41:36.765678 2031 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9347aa43aa114de5bcdd634f0b35f252 [term 1 LEADER]: Becoming Leader. State: Replica: 9347aa43aa114de5bcdd634f0b35f252, State: Running, Role: LEADER
I20260929 18:41:36.766026 2031 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9347aa43aa114de5bcdd634f0b35f252 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9347aa43aa114de5bcdd634f0b35f252" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32891 } }
I20260929 18:41:36.766463 2021 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9347aa43aa114de5bcdd634f0b35f252 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:41:36.767513 2033 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9347aa43aa114de5bcdd634f0b35f252 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9347aa43aa114de5bcdd634f0b35f252. Latest consensus state: current_term: 1 leader_uuid: "9347aa43aa114de5bcdd634f0b35f252" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9347aa43aa114de5bcdd634f0b35f252" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32891 } } }
I20260929 18:41:36.767643 2033 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9347aa43aa114de5bcdd634f0b35f252 [sys.catalog]: This master's current role is: LEADER
I20260929 18:41:36.768045 2040 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:41:36.768013 2032 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9347aa43aa114de5bcdd634f0b35f252 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9347aa43aa114de5bcdd634f0b35f252" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9347aa43aa114de5bcdd634f0b35f252" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32891 } } }
I20260929 18:41:36.768147 2032 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9347aa43aa114de5bcdd634f0b35f252 [sys.catalog]: This master's current role is: LEADER
I20260929 18:41:36.771288 2040 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:41:36.777067 2040 catalog_manager.cc:1347] Generated new cluster ID: 8205d21a5d444e80895c543def8a603b
I20260929 18:41:36.777138 2040 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:41:36.796294 2040 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:41:36.797290 2040 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:41:36.802704 2040 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9347aa43aa114de5bcdd634f0b35f252: Generated new TSK 0
I20260929 18:41:36.803395 2040 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:41:36.810487 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 1953
2026-09-29T18:41:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1406 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790707036041078-13712-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:41:36 dist-test-slave-j1xs krb5kdc[2052](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:41:36 dist-test-slave-j1xs krb5kdc[2052](info): set up 2 sockets
Sep 29 18:41:36 dist-test-slave-j1xs krb5kdc[2052](info): commencing operation
krb5kdc: starting...
W20260929 18:41:37.858054 13712 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790707036041078-13712-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790707036041078-13712-0/krb5kdc/test-user.keytab.
Sep 29 18:41:37 dist-test-slave-j1xs krb5kdc[2052](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707297, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:41:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:41:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790707036041078-13712-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790707036041078-13712-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 18:41:37.982031 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790707036041078-13712-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790707036041078-13712-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790707036041078-13712-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37027
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790707036041078-13712-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37027
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790707036041078-13712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790707036041078-13712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790707036041078-13712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:41:38.091104 2065 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:41:38.091418 2065 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:41:38.091492 2065 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:41:38.095645 2065 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:41:38.095762 2065 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:41:38.095798 2065 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:41:38.095856 2065 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:41:38.095896 2065 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:41:38.101424 2065 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790707036041078-13712-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790707036041078-13712-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37027
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790707036041078-13712-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37027
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2065
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790707036041078-13712-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:41:38.102634 2065 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:41:38.103705 2065 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:41:38.112177 2071 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:41:38.112177 2070 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:41:38.112179 2073 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:41:38.112594 2065 server_base.cc:1034] running on GCE node
I20260929 18:41:38.113128 2065 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:41:38.114347 2065 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:41:38.115537 2065 hybrid_clock.cc:648] HybridClock initialized: now 1790707298115507 us; error 45 us; skew 500 ppm
Sep 29 18:41:38 dist-test-slave-j1xs krb5kdc[2052](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707298, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:41:38.118593 2065 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 18:41:38.119539 2065 webserver.cc:460] Webserver started at http://127.0.0.1:35335/ using document root <none> and password file <none>
I20260929 18:41:38.120110 2065 fs_manager.cc:359] Metadata directory not provided
I20260929 18:41:38.120209 2065 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:41:38.120450 2065 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:41:38.122632 2065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790707036041078-13712-0/minicluster-data/master-0/data/instance:
uuid: "3f8f7334b4d745d3a81bf4032de8335f"
format_stamp: "Formatted at 2026-09-29 18:41:38 on dist-test-slave-j1xs"
server_key: "4d147ea995de4e86f83ec5633bf29fd8"
server_key_iv: "a32ff6acfe143a2f7f3797eecad13740"
server_key_version: "encryptionkey@0"
I20260929 18:41:38.123208 2065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790707036041078-13712-0/minicluster-data/master-0/wal/instance:
uuid: "3f8f7334b4d745d3a81bf4032de8335f"
format_stamp: "Formatted at 2026-09-29 18:41:38 on dist-test-slave-j1xs"
server_key: "4d147ea995de4e86f83ec5633bf29fd8"
server_key_iv: "a32ff6acfe143a2f7f3797eecad13740"
server_key_version: "encryptionkey@0"
I20260929 18:41:38.127159 2065 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 18:41:38.129591 2080 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:41:38.131047 2065 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:41:38.131192 2065 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790707036041078-13712-0/minicluster-data/master-0/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790707036041078-13712-0/minicluster-data/master-0/wal
uuid: "3f8f7334b4d745d3a81bf4032de8335f"
format_stamp: "Formatted at 2026-09-29 18:41:38 on dist-test-slave-j1xs"
server_key: "4d147ea995de4e86f83ec5633bf29fd8"
server_key_iv: "a32ff6acfe143a2f7f3797eecad13740"
server_key_version: "encryptionkey@0"
I20260929 18:41:38.131301 2065 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790707036041078-13712-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790707036041078-13712-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790707036041078-13712-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:41:38.136034 2065 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:41:38.139062 2065 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:41:38.139274 2065 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:41:38.145838 2065 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37027
I20260929 18:41:38.145928 2132 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37027 every 8 connection(s)
I20260929 18:41:38.146770 2065 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
I20260929 18:41:38.148152 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 2065
I20260929 18:41:38.148337 13712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790707036041078-13712-0/minicluster-data/master-0/wal/instance
I20260929 18:41:38.148639 13712 external_mini_cluster.cc:1428] Setting key 673e5483bff464acd214ef4911d8b5f2
I20260929 18:41:38.150017 2133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:41:38.153972 2136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:38.149991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52943 (local address 127.0.0.1:37027)
0929 18:41:38.150425 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0929 18:41:38.150439 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 18:41:38.150473 (+ 34us) server_negotiation.cc:407] Connection header received
0929 18:41:38.151084 (+ 611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:41:38.151099 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:41:38.151360 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:41:38.151667 (+ 307us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:41:38.152595 (+ 928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:41:38.152600 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:41:38.152611 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:41:38.152622 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:41:38.152671 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:41:38.152689 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:41:38.152796 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 18:41:38.152903 (+ 107us) server_negotiation.cc:299] Negotiation successful
0929 18:41:38.153059 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":150,"threads_started":1}
I20260929 18:41:38.156328 2133 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f8f7334b4d745d3a81bf4032de8335f: Bootstrap starting.
I20260929 18:41:38.157094 13712 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:41:38.158888 2133 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3f8f7334b4d745d3a81bf4032de8335f: Neither blocks nor log segments found. Creating new log.
I20260929 18:41:38.159795 2133 log.cc:824] T 00000000000000000000000000000000 P 3f8f7334b4d745d3a81bf4032de8335f: Log is configured to *not* fsync() on all Append() calls
I20260929 18:41:38.160948 2136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:38.159271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50339 (local address 127.0.0.1:37027)
0929 18:41:38.159403 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 18:41:38.159408 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:41:38.159622 (+ 214us) server_negotiation.cc:407] Connection header received
0929 18:41:38.159754 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:41:38.159756 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:41:38.159786 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:41:38.159850 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:41:38.160568 (+ 718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:41:38.160571 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:41:38.160573 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:41:38.160584 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:41:38.160607 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:41:38.160617 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:41:38.160711 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0929 18:41:38.160786 (+ 75us) server_negotiation.cc:299] Negotiation successful
0929 18:41:38.160827 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 18:41:38.162026 2133 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f8f7334b4d745d3a81bf4032de8335f: No bootstrap required, opened a new log
I20260929 18:41:38.165709 2133 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3f8f7334b4d745d3a81bf4032de8335f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f8f7334b4d745d3a81bf4032de8335f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37027 } }
I20260929 18:41:38.166009 2133 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3f8f7334b4d745d3a81bf4032de8335f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:41:38.166086 2133 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3f8f7334b4d745d3a81bf4032de8335f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f8f7334b4d745d3a81bf4032de8335f, State: Initialized, Role: FOLLOWER
I20260929 18:41:38.166559 2133 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3f8f7334b4d745d3a81bf4032de8335f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f8f7334b4d745d3a81bf4032de8335f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37027 } }
I20260929 18:41:38.166685 2133 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3f8f7334b4d745d3a81bf4032de8335f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:41:38.166725 2133 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3f8f7334b4d745d3a81bf4032de8335f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:41:38.166796 2133 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3f8f7334b4d745d3a81bf4032de8335f [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:41:38.167631 2133 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3f8f7334b4d745d3a81bf4032de8335f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f8f7334b4d745d3a81bf4032de8335f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37027 } }
I20260929 18:41:38.167915 2133 leader_election.cc:302] T 00000000000000000000000000000000 P 3f8f7334b4d745d3a81bf4032de8335f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3f8f7334b4d745d3a81bf4032de8335f; no voters:
I20260929 18:41:38.168160 2133 leader_election.cc:288] T 00000000000000000000000000000000 P 3f8f7334b4d745d3a81bf4032de8335f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:41:38.168279 2143 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3f8f7334b4d745d3a81bf4032de8335f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:41:38.168545 2143 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3f8f7334b4d745d3a81bf4032de8335f [term 1 LEADER]: Becoming Leader. State: Replica: 3f8f7334b4d745d3a81bf4032de8335f, State: Running, Role: LEADER
I20260929 18:41:38.168829 2143 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3f8f7334b4d745d3a81bf4032de8335f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f8f7334b4d745d3a81bf4032de8335f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37027 } }
I20260929 18:41:38.169338 2133 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3f8f7334b4d745d3a81bf4032de8335f [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:41:38.170565 2145 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f8f7334b4d745d3a81bf4032de8335f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3f8f7334b4d745d3a81bf4032de8335f. Latest consensus state: current_term: 1 leader_uuid: "3f8f7334b4d745d3a81bf4032de8335f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f8f7334b4d745d3a81bf4032de8335f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37027 } } }
I20260929 18:41:38.170583 2144 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f8f7334b4d745d3a81bf4032de8335f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3f8f7334b4d745d3a81bf4032de8335f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f8f7334b4d745d3a81bf4032de8335f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37027 } } }
I20260929 18:41:38.170711 2145 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f8f7334b4d745d3a81bf4032de8335f [sys.catalog]: This master's current role is: LEADER
I20260929 18:41:38.170715 2144 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f8f7334b4d745d3a81bf4032de8335f [sys.catalog]: This master's current role is: LEADER
I20260929 18:41:38.171067 2152 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:41:38.173719 2152 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:41:38.179172 2152 catalog_manager.cc:1347] Generated new cluster ID: 200417ff50ef420bbd1ff70feb814564
I20260929 18:41:38.179250 2152 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:41:38.196540 2152 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:41:38.196844 2152 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:41:38.209239 2152 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3f8f7334b4d745d3a81bf4032de8335f: Generated new TSK 0
I20260929 18:41:38.209980 2152 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:41:38.221952 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 2065
2026-09-29T18:41:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1410 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790707036041078-13712-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:41:38 dist-test-slave-j1xs krb5kdc[2164](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:41:38 dist-test-slave-j1xs krb5kdc[2164](info): set up 2 sockets
Sep 29 18:41:38 dist-test-slave-j1xs krb5kdc[2164](info): commencing operation
krb5kdc: starting...
W20260929 18:41:39.262128 13712 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790707036041078-13712-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790707036041078-13712-0/krb5kdc/test-user.keytab.
Sep 29 18:41:39 dist-test-slave-j1xs krb5kdc[2164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707299, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:41:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:41:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790707036041078-13712-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790707036041078-13712-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 18:41:39.382964 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790707036041078-13712-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790707036041078-13712-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790707036041078-13712-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45501
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790707036041078-13712-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:32821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45501
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790707036041078-13712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790707036041078-13712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790707036041078-13712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:41:39.488703 2177 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:41:39.488955 2177 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:41:39.488996 2177 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:41:39.492839 2177 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:41:39.492888 2177 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:41:39.492908 2177 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:41:39.492923 2177 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:41:39.492937 2177 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:41:39.498148 2177 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:32821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790707036041078-13712-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790707036041078-13712-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45501
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790707036041078-13712-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45501
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2177
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790707036041078-13712-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:41:39.499084 2177 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:41:39.500002 2177 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:41:39.508224 2183 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:41:39.508224 2185 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:41:39.508437 2177 server_base.cc:1034] running on GCE node
W20260929 18:41:39.508246 2182 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:41:39.509040 2177 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:41:39.510196 2177 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:41:39.511405 2177 hybrid_clock.cc:648] HybridClock initialized: now 1790707299511389 us; error 30 us; skew 500 ppm
Sep 29 18:41:39 dist-test-slave-j1xs krb5kdc[2164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707299, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:41:39.514178 2177 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 18:41:39.515067 2177 webserver.cc:460] Webserver started at http://127.0.0.1:35063/ using document root <none> and password file <none>
I20260929 18:41:39.515611 2177 fs_manager.cc:359] Metadata directory not provided
I20260929 18:41:39.515702 2177 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:41:39.515937 2177 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:41:39.517969 2177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790707036041078-13712-0/minicluster-data/master-0/data/instance:
uuid: "d5a184da6b514b1c9ccadf28967d3204"
format_stamp: "Formatted at 2026-09-29 18:41:39 on dist-test-slave-j1xs"
server_key: "70bb54c07d47353bfa23f4984bd973bf"
server_key_iv: "095a97d288ed6fa72e9ec4bb0c44382e"
server_key_version: "encryptionkey@0"
I20260929 18:41:39.518567 2177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790707036041078-13712-0/minicluster-data/master-0/wal/instance:
uuid: "d5a184da6b514b1c9ccadf28967d3204"
format_stamp: "Formatted at 2026-09-29 18:41:39 on dist-test-slave-j1xs"
server_key: "70bb54c07d47353bfa23f4984bd973bf"
server_key_iv: "095a97d288ed6fa72e9ec4bb0c44382e"
server_key_version: "encryptionkey@0"
I20260929 18:41:39.522107 2177 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 18:41:39.524462 2192 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:41:39.525636 2177 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:41:39.525774 2177 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790707036041078-13712-0/minicluster-data/master-0/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790707036041078-13712-0/minicluster-data/master-0/wal
uuid: "d5a184da6b514b1c9ccadf28967d3204"
format_stamp: "Formatted at 2026-09-29 18:41:39 on dist-test-slave-j1xs"
server_key: "70bb54c07d47353bfa23f4984bd973bf"
server_key_iv: "095a97d288ed6fa72e9ec4bb0c44382e"
server_key_version: "encryptionkey@0"
I20260929 18:41:39.525878 2177 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790707036041078-13712-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790707036041078-13712-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790707036041078-13712-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:41:39.540392 2177 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:41:39.543431 2177 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:41:39.543632 2177 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:41:39.550876 2177 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45501
I20260929 18:41:39.550899 2244 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45501 every 8 connection(s)
I20260929 18:41:39.551817 2177 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
I20260929 18:41:39.554666 2245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:41:39.558967 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 2177
I20260929 18:41:39.559280 13712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790707036041078-13712-0/minicluster-data/master-0/wal/instance
I20260929 18:41:39.559944 13712 external_mini_cluster.cc:1428] Setting key 5a917eea576d1f11d009deb261f35995
I20260929 18:41:39.560590 2245 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5a184da6b514b1c9ccadf28967d3204: Bootstrap starting.
I20260929 18:41:39.563248 2245 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d5a184da6b514b1c9ccadf28967d3204: Neither blocks nor log segments found. Creating new log.
I20260929 18:41:39.564034 2245 log.cc:824] T 00000000000000000000000000000000 P d5a184da6b514b1c9ccadf28967d3204: Log is configured to *not* fsync() on all Append() calls
I20260929 18:41:39.566087 2245 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5a184da6b514b1c9ccadf28967d3204: No bootstrap required, opened a new log
Sep 29 18:41:39 dist-test-slave-j1xs krb5kdc[2164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707299, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 18:41:39.570586 2245 raft_consensus.cc:348] T 00000000000000000000000000000000 P d5a184da6b514b1c9ccadf28967d3204 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5a184da6b514b1c9ccadf28967d3204" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45501 } }
I20260929 18:41:39.570847 2245 raft_consensus.cc:374] T 00000000000000000000000000000000 P d5a184da6b514b1c9ccadf28967d3204 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:41:39.570912 2245 raft_consensus.cc:729] T 00000000000000000000000000000000 P d5a184da6b514b1c9ccadf28967d3204 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5a184da6b514b1c9ccadf28967d3204, State: Initialized, Role: FOLLOWER
I20260929 18:41:39.571405 2245 consensus_queue.cc:260] T 00000000000000000000000000000000 P d5a184da6b514b1c9ccadf28967d3204 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5a184da6b514b1c9ccadf28967d3204" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45501 } }
I20260929 18:41:39.571578 2245 raft_consensus.cc:388] T 00000000000000000000000000000000 P d5a184da6b514b1c9ccadf28967d3204 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:41:39.571652 2245 raft_consensus.cc:482] T 00000000000000000000000000000000 P d5a184da6b514b1c9ccadf28967d3204 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:41:39.571761 2245 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d5a184da6b514b1c9ccadf28967d3204 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:41:39.572624 2245 raft_consensus.cc:504] T 00000000000000000000000000000000 P d5a184da6b514b1c9ccadf28967d3204 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5a184da6b514b1c9ccadf28967d3204" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45501 } }
I20260929 18:41:39.572906 2245 leader_election.cc:302] T 00000000000000000000000000000000 P d5a184da6b514b1c9ccadf28967d3204 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d5a184da6b514b1c9ccadf28967d3204; no voters:
I20260929 18:41:39.573266 2245 leader_election.cc:288] T 00000000000000000000000000000000 P d5a184da6b514b1c9ccadf28967d3204 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:41:39.573349 2250 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d5a184da6b514b1c9ccadf28967d3204 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:41:39.573536 2250 raft_consensus.cc:686] T 00000000000000000000000000000000 P d5a184da6b514b1c9ccadf28967d3204 [term 1 LEADER]: Becoming Leader. State: Replica: d5a184da6b514b1c9ccadf28967d3204, State: Running, Role: LEADER
I20260929 18:41:39.573817 2250 consensus_queue.cc:237] T 00000000000000000000000000000000 P d5a184da6b514b1c9ccadf28967d3204 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5a184da6b514b1c9ccadf28967d3204" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45501 } }
I20260929 18:41:39.574538 2245 sys_catalog.cc:564] T 00000000000000000000000000000000 P d5a184da6b514b1c9ccadf28967d3204 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:41:39.575395 2251 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5a184da6b514b1c9ccadf28967d3204 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d5a184da6b514b1c9ccadf28967d3204. Latest consensus state: current_term: 1 leader_uuid: "d5a184da6b514b1c9ccadf28967d3204" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5a184da6b514b1c9ccadf28967d3204" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45501 } } }
I20260929 18:41:39.575547 2251 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5a184da6b514b1c9ccadf28967d3204 [sys.catalog]: This master's current role is: LEADER
I20260929 18:41:39.575759 2252 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5a184da6b514b1c9ccadf28967d3204 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d5a184da6b514b1c9ccadf28967d3204" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5a184da6b514b1c9ccadf28967d3204" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45501 } } }
I20260929 18:41:39.575870 2252 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5a184da6b514b1c9ccadf28967d3204 [sys.catalog]: This master's current role is: LEADER
I20260929 18:41:39.576123 2259 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:41:39.577064 2248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:39.561829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.43:33027 (local address 127.0.0.1:45501)
0929 18:41:39.562502 (+ 673us) server_negotiation.cc:206] Beginning negotiation
0929 18:41:39.562532 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0929 18:41:39.562583 (+ 51us) server_negotiation.cc:407] Connection header received
0929 18:41:39.563641 (+ 1058us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:41:39.563665 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:41:39.564034 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:41:39.564364 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:41:39.565198 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:39.566247 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:39.567390 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:39.567839 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:39.567959 (+ 120us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:39.570829 (+ 2870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:41:39.570835 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:41:39.570852 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:41:39.570888 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:41:39.573045 (+ 2157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:41:39.573791 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:41:39.573797 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:41:39.573803 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:41:39.573981 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:41:39.574413 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:41:39.574419 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:41:39.574422 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:41:39.574987 (+ 565us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:41:39.575180 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0929 18:41:39.575793 (+ 613us) server_negotiation.cc:299] Negotiation successful
0929 18:41:39.576064 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":482,"thread_start_us":238,"threads_started":1}
I20260929 18:41:39.579325 2259 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:41:39.580718 13712 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:41:39.586335 2259 catalog_manager.cc:1347] Generated new cluster ID: 87a9f54adee84279a7d595346e974bf2
I20260929 18:41:39.586410 2259 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:41:39.592353 2248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:39.582999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.43:35007 (local address 127.0.0.1:45501)
0929 18:41:39.583112 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0929 18:41:39.583117 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:41:39.583695 (+ 578us) server_negotiation.cc:407] Connection header received
0929 18:41:39.583743 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:41:39.583746 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:41:39.583798 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:41:39.583917 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:41:39.584782 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:39.585363 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:39.586507 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:39.586860 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:39.586969 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:39.588700 (+ 1731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:41:39.588704 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:41:39.588707 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:41:39.588732 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:41:39.590462 (+ 1730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:41:39.591010 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:41:39.591014 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:41:39.591016 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:41:39.591098 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:41:39.591465 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:41:39.591468 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:41:39.591470 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:41:39.591673 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:41:39.591828 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0929 18:41:39.592097 (+ 269us) server_negotiation.cc:299] Negotiation successful
0929 18:41:39.592192 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 18:41:39.595396 2259 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:41:39.596314 2259 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:41:39.602707 2259 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d5a184da6b514b1c9ccadf28967d3204: Generated new TSK 0
I20260929 18:41:39.603334 2259 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:41:39.607065 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 2177
W20260929 18:41:39.612457 2162 connection.cc:535] client connection to 127.0.0.1:45501 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45501): Connection reset by peer (error 104)
2026-09-29T18:41:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1385 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790707036041078-13712-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:41:39 dist-test-slave-j1xs krb5kdc[2276](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:41:39 dist-test-slave-j1xs krb5kdc[2276](info): set up 2 sockets
Sep 29 18:41:39 dist-test-slave-j1xs krb5kdc[2276](info): commencing operation
krb5kdc: starting...
W20260929 18:41:40.666086 13712 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790707036041078-13712-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790707036041078-13712-0/krb5kdc/test-user.keytab.
Sep 29 18:41:40 dist-test-slave-j1xs krb5kdc[2276](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707300, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:41:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:41:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790707036041078-13712-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790707036041078-13712-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 18:41:40.785331 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790707036041078-13712-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790707036041078-13712-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790707036041078-13712-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33089
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790707036041078-13712-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33089
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790707036041078-13712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790707036041078-13712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790707036041078-13712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:41:40.894605 2289 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:41:40.894882 2289 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:41:40.894955 2289 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:41:40.898913 2289 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:41:40.898999 2289 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:41:40.899045 2289 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:41:40.899097 2289 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:41:40.899137 2289 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:41:40.904356 2289 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790707036041078-13712-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790707036041078-13712-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33089
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790707036041078-13712-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33089
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2289
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790707036041078-13712-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:41:40.905372 2289 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:41:40.906536 2289 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:41:40.914932 2295 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:41:40.914966 2294 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:41:40.914950 2297 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:41:40.915342 2289 server_base.cc:1034] running on GCE node
I20260929 18:41:40.915915 2289 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:41:40.917032 2289 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:41:40.918233 2289 hybrid_clock.cc:648] HybridClock initialized: now 1790707300918210 us; error 32 us; skew 500 ppm
Sep 29 18:41:40 dist-test-slave-j1xs krb5kdc[2276](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707300, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:41:40.924286 2289 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 18:41:40.925163 2289 webserver.cc:460] Webserver started at http://127.0.0.1:43709/ using document root <none> and password file <none>
I20260929 18:41:40.925722 2289 fs_manager.cc:359] Metadata directory not provided
I20260929 18:41:40.925813 2289 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:41:40.926102 2289 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:41:40.928128 2289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790707036041078-13712-0/minicluster-data/master-0/data/instance:
uuid: "1598dcdc834b44fba8dbbbba3081b684"
format_stamp: "Formatted at 2026-09-29 18:41:40 on dist-test-slave-j1xs"
server_key: "5d8042d8484f338e30ab4e42b01d4134"
server_key_iv: "f5b2ffe2c646fb6d5ef051cd4c57cd5f"
server_key_version: "encryptionkey@0"
I20260929 18:41:40.928695 2289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790707036041078-13712-0/minicluster-data/master-0/wal/instance:
uuid: "1598dcdc834b44fba8dbbbba3081b684"
format_stamp: "Formatted at 2026-09-29 18:41:40 on dist-test-slave-j1xs"
server_key: "5d8042d8484f338e30ab4e42b01d4134"
server_key_iv: "f5b2ffe2c646fb6d5ef051cd4c57cd5f"
server_key_version: "encryptionkey@0"
I20260929 18:41:40.932554 2289 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260929 18:41:40.935367 2304 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:41:40.936546 2289 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 18:41:40.936679 2289 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790707036041078-13712-0/minicluster-data/master-0/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790707036041078-13712-0/minicluster-data/master-0/wal
uuid: "1598dcdc834b44fba8dbbbba3081b684"
format_stamp: "Formatted at 2026-09-29 18:41:40 on dist-test-slave-j1xs"
server_key: "5d8042d8484f338e30ab4e42b01d4134"
server_key_iv: "f5b2ffe2c646fb6d5ef051cd4c57cd5f"
server_key_version: "encryptionkey@0"
I20260929 18:41:40.936791 2289 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790707036041078-13712-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790707036041078-13712-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790707036041078-13712-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:41:40.957265 2289 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:41:40.960273 2289 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:41:40.960490 2289 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:41:40.967762 2289 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33089
I20260929 18:41:40.967787 2356 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33089 every 8 connection(s)
I20260929 18:41:40.968670 2289 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
I20260929 18:41:40.971347 2357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:41:40.971649 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 2289
I20260929 18:41:40.971805 13712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790707036041078-13712-0/minicluster-data/master-0/wal/instance
I20260929 18:41:40.972090 13712 external_mini_cluster.cc:1428] Setting key 77aa68f2626519a41a8164689a376b1e
I20260929 18:41:40.978446 2357 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1598dcdc834b44fba8dbbbba3081b684: Bootstrap starting.
I20260929 18:41:40.981153 2357 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1598dcdc834b44fba8dbbbba3081b684: Neither blocks nor log segments found. Creating new log.
I20260929 18:41:40.982076 2357 log.cc:824] T 00000000000000000000000000000000 P 1598dcdc834b44fba8dbbbba3081b684: Log is configured to *not* fsync() on all Append() calls
I20260929 18:41:40.982375 2360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:40.973504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.43:43411 (local address 127.0.0.1:33089)
0929 18:41:40.973967 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0929 18:41:40.973985 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 18:41:40.974027 (+ 42us) server_negotiation.cc:407] Connection header received
0929 18:41:40.974741 (+ 714us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:41:40.974763 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:41:40.975202 (+ 439us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:41:40.975564 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:41:40.975967 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:40.977469 (+ 1502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:40.978562 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:40.979303 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:40.979468 (+ 165us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:40.980364 (+ 896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:41:40.980370 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:41:40.980385 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:41:40.980884 (+ 499us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:41:40.980949 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:41:40.980981 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:41:40.981137 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0929 18:41:40.981287 (+ 150us) server_negotiation.cc:299] Negotiation successful
0929 18:41:40.981428 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":113,"threads_started":1}
I20260929 18:41:40.984210 2357 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1598dcdc834b44fba8dbbbba3081b684: No bootstrap required, opened a new log
I20260929 18:41:40.985450 13712 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:41:40.988849 2357 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1598dcdc834b44fba8dbbbba3081b684 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1598dcdc834b44fba8dbbbba3081b684" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33089 } }
I20260929 18:41:40.989109 2357 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1598dcdc834b44fba8dbbbba3081b684 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:41:40.989171 2357 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1598dcdc834b44fba8dbbbba3081b684 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1598dcdc834b44fba8dbbbba3081b684, State: Initialized, Role: FOLLOWER
I20260929 18:41:40.989629 2357 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1598dcdc834b44fba8dbbbba3081b684 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1598dcdc834b44fba8dbbbba3081b684" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33089 } }
I20260929 18:41:40.989776 2357 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1598dcdc834b44fba8dbbbba3081b684 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:41:40.989837 2357 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1598dcdc834b44fba8dbbbba3081b684 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:41:40.989974 2357 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1598dcdc834b44fba8dbbbba3081b684 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:41:40.991101 2357 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1598dcdc834b44fba8dbbbba3081b684 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1598dcdc834b44fba8dbbbba3081b684" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33089 } }
I20260929 18:41:40.991472 2357 leader_election.cc:302] T 00000000000000000000000000000000 P 1598dcdc834b44fba8dbbbba3081b684 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1598dcdc834b44fba8dbbbba3081b684; no voters:
I20260929 18:41:40.991818 2357 leader_election.cc:288] T 00000000000000000000000000000000 P 1598dcdc834b44fba8dbbbba3081b684 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:41:40.991858 2360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:40.987666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.43:56797 (local address 127.0.0.1:33089)
0929 18:41:40.987791 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0929 18:41:40.987795 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:41:40.987975 (+ 180us) server_negotiation.cc:407] Connection header received
0929 18:41:40.988087 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:41:40.988089 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:41:40.988127 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:41:40.988200 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:41:40.988476 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:40.988964 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:40.990108 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:40.990364 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:40.990447 (+ 83us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:40.991328 (+ 881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:41:40.991332 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:41:40.991335 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:41:40.991347 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:41:40.991379 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:41:40.991390 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:41:40.991467 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 18:41:40.991665 (+ 198us) server_negotiation.cc:299] Negotiation successful
0929 18:41:40.991704 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 18:41:40.991918 2367 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1598dcdc834b44fba8dbbbba3081b684 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:41:40.992372 2367 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1598dcdc834b44fba8dbbbba3081b684 [term 1 LEADER]: Becoming Leader. State: Replica: 1598dcdc834b44fba8dbbbba3081b684, State: Running, Role: LEADER
I20260929 18:41:40.992650 2367 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1598dcdc834b44fba8dbbbba3081b684 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1598dcdc834b44fba8dbbbba3081b684" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33089 } }
I20260929 18:41:40.993105 2357 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1598dcdc834b44fba8dbbbba3081b684 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:41:40.994295 2368 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1598dcdc834b44fba8dbbbba3081b684 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1598dcdc834b44fba8dbbbba3081b684" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1598dcdc834b44fba8dbbbba3081b684" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33089 } } }
I20260929 18:41:40.994423 2368 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1598dcdc834b44fba8dbbbba3081b684 [sys.catalog]: This master's current role is: LEADER
I20260929 18:41:40.994727 2369 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1598dcdc834b44fba8dbbbba3081b684 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1598dcdc834b44fba8dbbbba3081b684. Latest consensus state: current_term: 1 leader_uuid: "1598dcdc834b44fba8dbbbba3081b684" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1598dcdc834b44fba8dbbbba3081b684" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33089 } } }
I20260929 18:41:40.994819 2376 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:41:40.994838 2369 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1598dcdc834b44fba8dbbbba3081b684 [sys.catalog]: This master's current role is: LEADER
I20260929 18:41:40.997687 2376 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:41:41.002919 2376 catalog_manager.cc:1347] Generated new cluster ID: 6b23bf9f2c9548d8a33c725c54a696e4
I20260929 18:41:41.002981 2376 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:41:41.019914 2376 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:41:41.020853 2376 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:41:41.030459 2376 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1598dcdc834b44fba8dbbbba3081b684: Generated new TSK 0
I20260929 18:41:41.031224 2376 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:41:41.040190 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 2289
W20260929 18:41:41.045660 2274 connection.cc:535] client connection to 127.0.0.1:33089 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33089): Connection reset by peer (error 104)
2026-09-29T18:41:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1433 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790707036041078-13712-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:41:41 dist-test-slave-j1xs krb5kdc[2388](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:41:41 dist-test-slave-j1xs krb5kdc[2388](info): set up 2 sockets
Sep 29 18:41:41 dist-test-slave-j1xs krb5kdc[2388](info): commencing operation
krb5kdc: starting...
W20260929 18:41:42.087396 13712 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790707036041078-13712-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790707036041078-13712-0/krb5kdc/test-user.keytab.
Sep 29 18:41:42 dist-test-slave-j1xs krb5kdc[2388](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707302, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:41:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:41:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790707036041078-13712-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790707036041078-13712-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 18:41:42.212850 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790707036041078-13712-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790707036041078-13712-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790707036041078-13712-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44529
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790707036041078-13712-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44529
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790707036041078-13712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790707036041078-13712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790707036041078-13712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:41:42.322448 2401 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:41:42.322779 2401 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:41:42.322850 2401 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:41:42.326879 2401 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:41:42.326979 2401 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:41:42.327019 2401 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:41:42.327057 2401 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:41:42.327095 2401 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:41:42.332355 2401 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790707036041078-13712-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790707036041078-13712-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44529
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790707036041078-13712-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44529
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2401
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790707036041078-13712-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:41:42.333422 2401 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:41:42.334475 2401 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:41:42.343251 2409 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:41:42.343252 2406 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:41:42.343305 2407 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:41:42.343608 2401 server_base.cc:1034] running on GCE node
I20260929 18:41:42.344148 2401 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:41:42.345135 2401 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:41:42.346294 2401 hybrid_clock.cc:648] HybridClock initialized: now 1790707302346279 us; error 25 us; skew 500 ppm
Sep 29 18:41:42 dist-test-slave-j1xs krb5kdc[2388](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707302, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:41:42.348985 2401 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 18:41:42.349758 2401 webserver.cc:460] Webserver started at http://127.0.0.1:41337/ using document root <none> and password file <none>
I20260929 18:41:42.350315 2401 fs_manager.cc:359] Metadata directory not provided
I20260929 18:41:42.350375 2401 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:41:42.350536 2401 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:41:42.352445 2401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790707036041078-13712-0/minicluster-data/master-0/data/instance:
uuid: "f6ae802e59aa42ca89e1b46632263bf0"
format_stamp: "Formatted at 2026-09-29 18:41:42 on dist-test-slave-j1xs"
server_key: "d69dfc07f583a5b426e7dfa2a6143572"
server_key_iv: "b4a691d86d7b8742cb7c87a163b4f8af"
server_key_version: "encryptionkey@0"
I20260929 18:41:42.352955 2401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790707036041078-13712-0/minicluster-data/master-0/wal/instance:
uuid: "f6ae802e59aa42ca89e1b46632263bf0"
format_stamp: "Formatted at 2026-09-29 18:41:42 on dist-test-slave-j1xs"
server_key: "d69dfc07f583a5b426e7dfa2a6143572"
server_key_iv: "b4a691d86d7b8742cb7c87a163b4f8af"
server_key_version: "encryptionkey@0"
I20260929 18:41:42.356541 2401 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260929 18:41:42.358999 2416 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:41:42.360167 2401 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:41:42.360303 2401 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790707036041078-13712-0/minicluster-data/master-0/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790707036041078-13712-0/minicluster-data/master-0/wal
uuid: "f6ae802e59aa42ca89e1b46632263bf0"
format_stamp: "Formatted at 2026-09-29 18:41:42 on dist-test-slave-j1xs"
server_key: "d69dfc07f583a5b426e7dfa2a6143572"
server_key_iv: "b4a691d86d7b8742cb7c87a163b4f8af"
server_key_version: "encryptionkey@0"
I20260929 18:41:42.360411 2401 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790707036041078-13712-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790707036041078-13712-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790707036041078-13712-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:41:42.364939 2401 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:41:42.368566 2401 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:41:42.368783 2401 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:41:42.375936 2401 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44529
I20260929 18:41:42.376024 2468 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44529 every 8 connection(s)
I20260929 18:41:42.376924 2401 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
I20260929 18:41:42.379212 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 2401
I20260929 18:41:42.379366 13712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790707036041078-13712-0/minicluster-data/master-0/wal/instance
I20260929 18:41:42.379659 13712 external_mini_cluster.cc:1428] Setting key fcb7d62ddfa98f9e0ccdf5888c3e1f58
I20260929 18:41:42.379911 2469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:41:42.385129 2472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:42.380980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.43:56207 (local address 127.0.0.1:44529)
0929 18:41:42.381406 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0929 18:41:42.381426 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 18:41:42.381466 (+ 40us) server_negotiation.cc:407] Connection header received
0929 18:41:42.382295 (+ 829us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:41:42.382318 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:41:42.382677 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:41:42.383031 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:41:42.383749 (+ 718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:41:42.383755 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:41:42.383769 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:41:42.383781 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:41:42.383844 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:41:42.383871 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:41:42.384014 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0929 18:41:42.384081 (+ 67us) server_negotiation.cc:299] Negotiation successful
0929 18:41:42.384227 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":125,"threads_started":1}
I20260929 18:41:42.387195 2469 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6ae802e59aa42ca89e1b46632263bf0: Bootstrap starting.
I20260929 18:41:42.387815 13712 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:41:42.389590 2469 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f6ae802e59aa42ca89e1b46632263bf0: Neither blocks nor log segments found. Creating new log.
I20260929 18:41:42.390573 2469 log.cc:824] T 00000000000000000000000000000000 P f6ae802e59aa42ca89e1b46632263bf0: Log is configured to *not* fsync() on all Append() calls
I20260929 18:41:42.391611 2472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:42.389783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.43:56289 (local address 127.0.0.1:44529)
0929 18:41:42.389936 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0929 18:41:42.389940 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:41:42.390139 (+ 199us) server_negotiation.cc:407] Connection header received
0929 18:41:42.390253 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:41:42.390256 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:41:42.390287 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:41:42.390352 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:41:42.391147 (+ 795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:41:42.391154 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:41:42.391160 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:41:42.391176 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:41:42.391205 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:41:42.391220 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:41:42.391313 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 18:41:42.391489 (+ 176us) server_negotiation.cc:299] Negotiation successful
0929 18:41:42.391525 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 18:41:42.392895 2469 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6ae802e59aa42ca89e1b46632263bf0: No bootstrap required, opened a new log
I20260929 18:41:42.396531 2469 raft_consensus.cc:348] T 00000000000000000000000000000000 P f6ae802e59aa42ca89e1b46632263bf0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6ae802e59aa42ca89e1b46632263bf0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44529 } }
I20260929 18:41:42.396765 2469 raft_consensus.cc:374] T 00000000000000000000000000000000 P f6ae802e59aa42ca89e1b46632263bf0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:41:42.396838 2469 raft_consensus.cc:729] T 00000000000000000000000000000000 P f6ae802e59aa42ca89e1b46632263bf0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6ae802e59aa42ca89e1b46632263bf0, State: Initialized, Role: FOLLOWER
I20260929 18:41:42.397305 2469 consensus_queue.cc:260] T 00000000000000000000000000000000 P f6ae802e59aa42ca89e1b46632263bf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6ae802e59aa42ca89e1b46632263bf0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44529 } }
I20260929 18:41:42.397432 2469 raft_consensus.cc:388] T 00000000000000000000000000000000 P f6ae802e59aa42ca89e1b46632263bf0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:41:42.397473 2469 raft_consensus.cc:482] T 00000000000000000000000000000000 P f6ae802e59aa42ca89e1b46632263bf0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:41:42.397614 2469 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f6ae802e59aa42ca89e1b46632263bf0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:41:42.398639 2469 raft_consensus.cc:504] T 00000000000000000000000000000000 P f6ae802e59aa42ca89e1b46632263bf0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6ae802e59aa42ca89e1b46632263bf0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44529 } }
I20260929 18:41:42.398988 2469 leader_election.cc:302] T 00000000000000000000000000000000 P f6ae802e59aa42ca89e1b46632263bf0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f6ae802e59aa42ca89e1b46632263bf0; no voters:
I20260929 18:41:42.399305 2469 leader_election.cc:288] T 00000000000000000000000000000000 P f6ae802e59aa42ca89e1b46632263bf0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:41:42.399533 2479 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f6ae802e59aa42ca89e1b46632263bf0 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:41:42.399808 2479 raft_consensus.cc:686] T 00000000000000000000000000000000 P f6ae802e59aa42ca89e1b46632263bf0 [term 1 LEADER]: Becoming Leader. State: Replica: f6ae802e59aa42ca89e1b46632263bf0, State: Running, Role: LEADER
I20260929 18:41:42.400103 2479 consensus_queue.cc:237] T 00000000000000000000000000000000 P f6ae802e59aa42ca89e1b46632263bf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6ae802e59aa42ca89e1b46632263bf0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44529 } }
I20260929 18:41:42.400601 2469 sys_catalog.cc:564] T 00000000000000000000000000000000 P f6ae802e59aa42ca89e1b46632263bf0 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:41:42.401718 2480 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6ae802e59aa42ca89e1b46632263bf0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f6ae802e59aa42ca89e1b46632263bf0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6ae802e59aa42ca89e1b46632263bf0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44529 } } }
I20260929 18:41:42.401847 2480 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6ae802e59aa42ca89e1b46632263bf0 [sys.catalog]: This master's current role is: LEADER
I20260929 18:41:42.402242 2481 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6ae802e59aa42ca89e1b46632263bf0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f6ae802e59aa42ca89e1b46632263bf0. Latest consensus state: current_term: 1 leader_uuid: "f6ae802e59aa42ca89e1b46632263bf0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6ae802e59aa42ca89e1b46632263bf0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44529 } } }
I20260929 18:41:42.402334 2481 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6ae802e59aa42ca89e1b46632263bf0 [sys.catalog]: This master's current role is: LEADER
I20260929 18:41:42.402429 2488 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:41:42.405490 2488 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:41:42.410818 2488 catalog_manager.cc:1347] Generated new cluster ID: 916bf26b30db44678b60f08239ff7d10
I20260929 18:41:42.410887 2488 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:41:42.423668 2488 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:41:42.424031 2488 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:41:42.436676 2488 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f6ae802e59aa42ca89e1b46632263bf0: Generated new TSK 0
I20260929 18:41:42.437400 2488 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:41:42.445122 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 2401
2026-09-29T18:41:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1404 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790707036041078-13712-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:41:42 dist-test-slave-j1xs krb5kdc[2500](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:41:42 dist-test-slave-j1xs krb5kdc[2500](info): set up 2 sockets
Sep 29 18:41:42 dist-test-slave-j1xs krb5kdc[2500](info): commencing operation
krb5kdc: starting...
W20260929 18:41:43.486148 13712 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790707036041078-13712-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790707036041078-13712-0/krb5kdc/test-user.keytab.
Sep 29 18:41:43 dist-test-slave-j1xs krb5kdc[2500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707303, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:41:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:41:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790707036041078-13712-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790707036041078-13712-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 18:41:43.607067 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790707036041078-13712-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790707036041078-13712-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790707036041078-13712-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40991
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790707036041078-13712-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40991
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790707036041078-13712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790707036041078-13712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790707036041078-13712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:41:43.715904 2513 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:41:43.716213 2513 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:41:43.716293 2513 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:41:43.720393 2513 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:41:43.720482 2513 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:41:43.720532 2513 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:41:43.720593 2513 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:41:43.720634 2513 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:41:43.726018 2513 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790707036041078-13712-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790707036041078-13712-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40991
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790707036041078-13712-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40991
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2513
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790707036041078-13712-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:41:43.727072 2513 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:41:43.728058 2513 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:41:43.736471 2521 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:41:43.736670 2513 server_base.cc:1034] running on GCE node
W20260929 18:41:43.736486 2519 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:41:43.736519 2518 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:41:43.737301 2513 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:41:43.738415 2513 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:41:43.739606 2513 hybrid_clock.cc:648] HybridClock initialized: now 1790707303739587 us; error 40 us; skew 500 ppm
Sep 29 18:41:43 dist-test-slave-j1xs krb5kdc[2500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707303, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:41:43.742355 2513 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 18:41:43.743207 2513 webserver.cc:460] Webserver started at http://127.0.0.1:46085/ using document root <none> and password file <none>
I20260929 18:41:43.743736 2513 fs_manager.cc:359] Metadata directory not provided
I20260929 18:41:43.743824 2513 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:41:43.744067 2513 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:41:43.746184 2513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790707036041078-13712-0/minicluster-data/master-0/data/instance:
uuid: "d2530572127e44808e18976140bb99a4"
format_stamp: "Formatted at 2026-09-29 18:41:43 on dist-test-slave-j1xs"
server_key: "b51e85b3105c879c5b4542b8840c02c0"
server_key_iv: "1d4df27ebaf6dff47b474d32445e6408"
server_key_version: "encryptionkey@0"
I20260929 18:41:43.746727 2513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790707036041078-13712-0/minicluster-data/master-0/wal/instance:
uuid: "d2530572127e44808e18976140bb99a4"
format_stamp: "Formatted at 2026-09-29 18:41:43 on dist-test-slave-j1xs"
server_key: "b51e85b3105c879c5b4542b8840c02c0"
server_key_iv: "1d4df27ebaf6dff47b474d32445e6408"
server_key_version: "encryptionkey@0"
I20260929 18:41:43.750351 2513 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:41:43.752642 2528 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:41:43.753803 2513 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:41:43.753957 2513 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790707036041078-13712-0/minicluster-data/master-0/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790707036041078-13712-0/minicluster-data/master-0/wal
uuid: "d2530572127e44808e18976140bb99a4"
format_stamp: "Formatted at 2026-09-29 18:41:43 on dist-test-slave-j1xs"
server_key: "b51e85b3105c879c5b4542b8840c02c0"
server_key_iv: "1d4df27ebaf6dff47b474d32445e6408"
server_key_version: "encryptionkey@0"
I20260929 18:41:43.754086 2513 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790707036041078-13712-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790707036041078-13712-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790707036041078-13712-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:41:43.765255 2513 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:41:43.768183 2513 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:41:43.768373 2513 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:41:43.775664 2513 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40991
I20260929 18:41:43.775688 2580 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40991 every 8 connection(s)
I20260929 18:41:43.776552 2513 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
I20260929 18:41:43.779104 2581 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:41:43.782974 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 2513
I20260929 18:41:43.783102 13712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790707036041078-13712-0/minicluster-data/master-0/wal/instance
I20260929 18:41:43.783371 13712 external_mini_cluster.cc:1428] Setting key 9f34af993a76adb6716f6892ae2628ea
I20260929 18:41:43.784683 2581 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2530572127e44808e18976140bb99a4: Bootstrap starting.
I20260929 18:41:43.786854 2581 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d2530572127e44808e18976140bb99a4: Neither blocks nor log segments found. Creating new log.
I20260929 18:41:43.787578 2581 log.cc:824] T 00000000000000000000000000000000 P d2530572127e44808e18976140bb99a4: Log is configured to *not* fsync() on all Append() calls
I20260929 18:41:43.789331 2581 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2530572127e44808e18976140bb99a4: No bootstrap required, opened a new log
Sep 29 18:41:43 dist-test-slave-j1xs krb5kdc[2500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707303, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 18:41:43.793597 2581 raft_consensus.cc:348] T 00000000000000000000000000000000 P d2530572127e44808e18976140bb99a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2530572127e44808e18976140bb99a4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40991 } }
I20260929 18:41:43.793934 2581 raft_consensus.cc:374] T 00000000000000000000000000000000 P d2530572127e44808e18976140bb99a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:41:43.794008 2581 raft_consensus.cc:729] T 00000000000000000000000000000000 P d2530572127e44808e18976140bb99a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2530572127e44808e18976140bb99a4, State: Initialized, Role: FOLLOWER
I20260929 18:41:43.794469 2581 consensus_queue.cc:260] T 00000000000000000000000000000000 P d2530572127e44808e18976140bb99a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2530572127e44808e18976140bb99a4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40991 } }
I20260929 18:41:43.794587 2581 raft_consensus.cc:388] T 00000000000000000000000000000000 P d2530572127e44808e18976140bb99a4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:41:43.794672 2581 raft_consensus.cc:482] T 00000000000000000000000000000000 P d2530572127e44808e18976140bb99a4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:41:43.794771 2581 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d2530572127e44808e18976140bb99a4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:41:43.795962 2581 raft_consensus.cc:504] T 00000000000000000000000000000000 P d2530572127e44808e18976140bb99a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2530572127e44808e18976140bb99a4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40991 } }
I20260929 18:41:43.796303 2581 leader_election.cc:302] T 00000000000000000000000000000000 P d2530572127e44808e18976140bb99a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d2530572127e44808e18976140bb99a4; no voters:
I20260929 18:41:43.796638 2581 leader_election.cc:288] T 00000000000000000000000000000000 P d2530572127e44808e18976140bb99a4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:41:43.796800 2586 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d2530572127e44808e18976140bb99a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:41:43.797050 2586 raft_consensus.cc:686] T 00000000000000000000000000000000 P d2530572127e44808e18976140bb99a4 [term 1 LEADER]: Becoming Leader. State: Replica: d2530572127e44808e18976140bb99a4, State: Running, Role: LEADER
I20260929 18:41:43.797405 2586 consensus_queue.cc:237] T 00000000000000000000000000000000 P d2530572127e44808e18976140bb99a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2530572127e44808e18976140bb99a4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40991 } }
I20260929 18:41:43.797868 2581 sys_catalog.cc:564] T 00000000000000000000000000000000 P d2530572127e44808e18976140bb99a4 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:41:43.798285 2584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:43.784670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.43:53243 (local address 127.0.0.1:40991)
0929 18:41:43.785070 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0929 18:41:43.785084 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 18:41:43.785117 (+ 33us) server_negotiation.cc:407] Connection header received
0929 18:41:43.785715 (+ 598us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:41:43.785730 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:41:43.786123 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:41:43.786444 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:41:43.787286 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:43.788321 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:43.789178 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:43.789698 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:43.789866 (+ 168us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:43.792776 (+ 2910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:41:43.792781 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:41:43.792794 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:41:43.792819 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:41:43.794854 (+ 2035us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:41:43.795386 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:41:43.795390 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:41:43.795394 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:41:43.795518 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:41:43.795872 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:41:43.795874 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:41:43.795875 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:41:43.796309 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:41:43.796524 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0929 18:41:43.796919 (+ 395us) server_negotiation.cc:299] Negotiation successful
0929 18:41:43.797351 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":127,"threads_started":1}
I20260929 18:41:43.798584 2586 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2530572127e44808e18976140bb99a4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d2530572127e44808e18976140bb99a4. Latest consensus state: current_term: 1 leader_uuid: "d2530572127e44808e18976140bb99a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2530572127e44808e18976140bb99a4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40991 } } }
I20260929 18:41:43.798688 2586 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2530572127e44808e18976140bb99a4 [sys.catalog]: This master's current role is: LEADER
I20260929 18:41:43.799060 2592 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:41:43.799312 2588 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2530572127e44808e18976140bb99a4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d2530572127e44808e18976140bb99a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2530572127e44808e18976140bb99a4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40991 } } }
I20260929 18:41:43.799409 2588 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2530572127e44808e18976140bb99a4 [sys.catalog]: This master's current role is: LEADER
I20260929 18:41:43.801563 13712 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:41:43.802508 2592 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:41:43.809517 2592 catalog_manager.cc:1347] Generated new cluster ID: 392d94796ded42f98847c5a015c2fdd3
I20260929 18:41:43.809933 2592 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:41:43.812749 2584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:43.803976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.43:41411 (local address 127.0.0.1:40991)
0929 18:41:43.804095 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0929 18:41:43.804099 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:41:43.804111 (+ 12us) server_negotiation.cc:407] Connection header received
0929 18:41:43.804199 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:41:43.804202 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:41:43.804243 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:41:43.804340 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:41:43.805167 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:43.806008 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:43.806965 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:43.807353 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:43.807485 (+ 132us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:43.809178 (+ 1693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:41:43.809181 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:41:43.809184 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:41:43.809209 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:41:43.810942 (+ 1733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:41:43.811434 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:41:43.811437 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:41:43.811439 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:41:43.811529 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:41:43.811859 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:41:43.811863 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:41:43.811864 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:41:43.812053 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:41:43.812206 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0929 18:41:43.812474 (+ 268us) server_negotiation.cc:299] Negotiation successful
0929 18:41:43.812580 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 18:41:43.831596 2592 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:41:43.832743 2592 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:41:43.838356 2592 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d2530572127e44808e18976140bb99a4: Generated new TSK 0
I20260929 18:41:43.839129 2592 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:41:43.842856 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 2513
W20260929 18:41:43.848301 2498 connection.cc:535] client connection to 127.0.0.1:40991 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40991): Connection reset by peer (error 104)
2026-09-29T18:41:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1398 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790707036041078-13712-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:41:43 dist-test-slave-j1xs krb5kdc[2612](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:41:43 dist-test-slave-j1xs krb5kdc[2612](info): set up 2 sockets
Sep 29 18:41:43 dist-test-slave-j1xs krb5kdc[2612](info): commencing operation
krb5kdc: starting...
W20260929 18:41:44.874336 13712 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790707036041078-13712-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790707036041078-13712-0/krb5kdc/test-user.keytab.
Sep 29 18:41:44 dist-test-slave-j1xs krb5kdc[2612](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707304, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:41:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:41:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790707036041078-13712-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790707036041078-13712-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 18:41:44.993184 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790707036041078-13712-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790707036041078-13712-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790707036041078-13712-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44899
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790707036041078-13712-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44899
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790707036041078-13712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790707036041078-13712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790707036041078-13712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:41:45.099874 2625 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:41:45.100145 2625 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:41:45.100189 2625 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:41:45.104198 2625 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:41:45.104264 2625 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:41:45.104283 2625 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:41:45.104301 2625 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:41:45.104316 2625 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:41:45.109464 2625 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790707036041078-13712-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790707036041078-13712-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44899
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790707036041078-13712-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44899
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2625
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790707036041078-13712-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:41:45.110563 2625 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:41:45.111509 2625 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:41:45.120163 2631 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:41:45.120153 2630 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:41:45.120288 2625 server_base.cc:1034] running on GCE node
W20260929 18:41:45.120209 2633 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:41:45.120936 2625 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:41:45.122062 2625 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:41:45.123238 2625 hybrid_clock.cc:648] HybridClock initialized: now 1790707305123212 us; error 40 us; skew 500 ppm
Sep 29 18:41:45 dist-test-slave-j1xs krb5kdc[2612](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707305, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:41:45.125850 2625 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 18:41:45.126751 2625 webserver.cc:460] Webserver started at http://127.0.0.1:34539/ using document root <none> and password file <none>
I20260929 18:41:45.127341 2625 fs_manager.cc:359] Metadata directory not provided
I20260929 18:41:45.127426 2625 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:41:45.127669 2625 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:41:45.130568 2625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790707036041078-13712-0/minicluster-data/master-0/data/instance:
uuid: "4480682c934049229e193257b94ef20b"
format_stamp: "Formatted at 2026-09-29 18:41:45 on dist-test-slave-j1xs"
server_key: "cae7e0fdf1a4faed6501a152497044e3"
server_key_iv: "5e004f50c591d0fef33e62484fe5bac9"
server_key_version: "encryptionkey@0"
I20260929 18:41:45.131295 2625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790707036041078-13712-0/minicluster-data/master-0/wal/instance:
uuid: "4480682c934049229e193257b94ef20b"
format_stamp: "Formatted at 2026-09-29 18:41:45 on dist-test-slave-j1xs"
server_key: "cae7e0fdf1a4faed6501a152497044e3"
server_key_iv: "5e004f50c591d0fef33e62484fe5bac9"
server_key_version: "encryptionkey@0"
I20260929 18:41:45.135926 2625 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 18:41:45.138273 2640 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:41:45.139520 2625 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:41:45.139652 2625 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790707036041078-13712-0/minicluster-data/master-0/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790707036041078-13712-0/minicluster-data/master-0/wal
uuid: "4480682c934049229e193257b94ef20b"
format_stamp: "Formatted at 2026-09-29 18:41:45 on dist-test-slave-j1xs"
server_key: "cae7e0fdf1a4faed6501a152497044e3"
server_key_iv: "5e004f50c591d0fef33e62484fe5bac9"
server_key_version: "encryptionkey@0"
I20260929 18:41:45.139765 2625 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790707036041078-13712-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790707036041078-13712-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790707036041078-13712-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:41:45.159080 2625 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:41:45.161823 2625 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:41:45.162045 2625 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:41:45.168853 2625 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44899
I20260929 18:41:45.168896 2692 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44899 every 8 connection(s)
I20260929 18:41:45.169765 2625 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
I20260929 18:41:45.172427 2693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:41:45.177510 2693 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4480682c934049229e193257b94ef20b: Bootstrap starting.
I20260929 18:41:45.178833 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 2625
I20260929 18:41:45.178965 13712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790707036041078-13712-0/minicluster-data/master-0/wal/instance
I20260929 18:41:45.179241 13712 external_mini_cluster.cc:1428] Setting key e0cdcad7db8ed0c74f2b8b78635a6ec9
I20260929 18:41:45.180307 2693 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4480682c934049229e193257b94ef20b: Neither blocks nor log segments found. Creating new log.
I20260929 18:41:45.181087 2693 log.cc:824] T 00000000000000000000000000000000 P 4480682c934049229e193257b94ef20b: Log is configured to *not* fsync() on all Append() calls
I20260929 18:41:45.182972 2693 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4480682c934049229e193257b94ef20b: No bootstrap required, opened a new log
I20260929 18:41:45.186448 2693 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4480682c934049229e193257b94ef20b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4480682c934049229e193257b94ef20b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44899 } }
I20260929 18:41:45.186647 2693 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4480682c934049229e193257b94ef20b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:41:45.186717 2693 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4480682c934049229e193257b94ef20b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4480682c934049229e193257b94ef20b, State: Initialized, Role: FOLLOWER
I20260929 18:41:45.187233 2693 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4480682c934049229e193257b94ef20b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4480682c934049229e193257b94ef20b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44899 } }
I20260929 18:41:45.187388 2693 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4480682c934049229e193257b94ef20b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:41:45.187490 2693 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4480682c934049229e193257b94ef20b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:41:45.187585 2693 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4480682c934049229e193257b94ef20b [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:41:45.187983 2696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:45.180298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.43:56383 (local address 127.0.0.1:44899)
0929 18:41:45.180746 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0929 18:41:45.180759 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 18:41:45.180792 (+ 33us) server_negotiation.cc:407] Connection header received
0929 18:41:45.181528 (+ 736us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:41:45.181549 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:41:45.182009 (+ 460us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:41:45.182322 (+ 313us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:41:45.182718 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:45.183705 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:45.184583 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:45.185051 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:45.185225 (+ 174us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:45.186177 (+ 952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:41:45.186181 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:41:45.186192 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:41:45.186560 (+ 368us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:41:45.186609 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:41:45.186655 (+ 46us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:41:45.186811 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0929 18:41:45.186990 (+ 179us) server_negotiation.cc:299] Negotiation successful
0929 18:41:45.187119 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":156,"threads_started":1}
I20260929 18:41:45.188539 2693 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4480682c934049229e193257b94ef20b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4480682c934049229e193257b94ef20b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44899 } }
I20260929 18:41:45.188895 2693 leader_election.cc:302] T 00000000000000000000000000000000 P 4480682c934049229e193257b94ef20b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4480682c934049229e193257b94ef20b; no voters:
I20260929 18:41:45.189245 2693 leader_election.cc:288] T 00000000000000000000000000000000 P 4480682c934049229e193257b94ef20b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:41:45.189337 2698 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4480682c934049229e193257b94ef20b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:41:45.189610 2698 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4480682c934049229e193257b94ef20b [term 1 LEADER]: Becoming Leader. State: Replica: 4480682c934049229e193257b94ef20b, State: Running, Role: LEADER
I20260929 18:41:45.189936 2698 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4480682c934049229e193257b94ef20b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4480682c934049229e193257b94ef20b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44899 } }
I20260929 18:41:45.190477 2693 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4480682c934049229e193257b94ef20b [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:41:45.191073 13712 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:41:45.191140 2699 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4480682c934049229e193257b94ef20b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4480682c934049229e193257b94ef20b. Latest consensus state: current_term: 1 leader_uuid: "4480682c934049229e193257b94ef20b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4480682c934049229e193257b94ef20b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44899 } } }
I20260929 18:41:45.191248 2699 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4480682c934049229e193257b94ef20b [sys.catalog]: This master's current role is: LEADER
I20260929 18:41:45.191887 2700 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4480682c934049229e193257b94ef20b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4480682c934049229e193257b94ef20b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4480682c934049229e193257b94ef20b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44899 } } }
I20260929 18:41:45.192008 2700 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4480682c934049229e193257b94ef20b [sys.catalog]: This master's current role is: LEADER
I20260929 18:41:45.192497 2705 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:41:45.195511 2705 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:41:45.198338 2696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:45.193592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.43:46787 (local address 127.0.0.1:44899)
0929 18:41:45.193713 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0929 18:41:45.193717 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:41:45.193926 (+ 209us) server_negotiation.cc:407] Connection header received
0929 18:41:45.194028 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:41:45.194031 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:41:45.194074 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:41:45.194154 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:41:45.194439 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:45.195200 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:45.196439 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:45.196782 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:45.196890 (+ 108us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:45.197830 (+ 940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:41:45.197834 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:41:45.197837 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:41:45.197850 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:41:45.197878 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:41:45.197917 (+ 39us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:41:45.198008 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 18:41:45.198155 (+ 147us) server_negotiation.cc:299] Negotiation successful
0929 18:41:45.198190 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 18:41:45.201162 2705 catalog_manager.cc:1347] Generated new cluster ID: 283d51b8b1264222b3f21342abec20a2
I20260929 18:41:45.201221 2705 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:41:45.226204 2705 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:41:45.227473 2705 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:41:45.234272 2705 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4480682c934049229e193257b94ef20b: Generated new TSK 0
I20260929 18:41:45.235030 2705 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:41:45.245486 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 2625
W20260929 18:41:45.250674 2610 connection.cc:535] client connection to 127.0.0.1:44899 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44899): Connection reset by peer (error 104)
2026-09-29T18:41:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1402 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790707036041078-13712-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:41:45 dist-test-slave-j1xs krb5kdc[2724](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:41:45 dist-test-slave-j1xs krb5kdc[2724](info): set up 2 sockets
Sep 29 18:41:45 dist-test-slave-j1xs krb5kdc[2724](info): commencing operation
krb5kdc: starting...
W20260929 18:41:46.282115 13712 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790707036041078-13712-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790707036041078-13712-0/krb5kdc/test-user.keytab.
Sep 29 18:41:46 dist-test-slave-j1xs krb5kdc[2724](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707306, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:41:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:41:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790707036041078-13712-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790707036041078-13712-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 18:41:46.400861 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790707036041078-13712-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790707036041078-13712-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790707036041078-13712-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45163
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790707036041078-13712-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45163
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790707036041078-13712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790707036041078-13712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790707036041078-13712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:41:46.508378 2737 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:41:46.508638 2737 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:41:46.508680 2737 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:41:46.512607 2737 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:41:46.512663 2737 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:41:46.512683 2737 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:41:46.512699 2737 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:41:46.512714 2737 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:41:46.518090 2737 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790707036041078-13712-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790707036041078-13712-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45163
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790707036041078-13712-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45163
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2737
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790707036041078-13712-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:41:46.519105 2737 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:41:46.520056 2737 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:41:46.528689 2743 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:41:46.528661 2745 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:41:46.528661 2742 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:41:46.528766 2737 server_base.cc:1034] running on GCE node
I20260929 18:41:46.529436 2737 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:41:46.530582 2737 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:41:46.531767 2737 hybrid_clock.cc:648] HybridClock initialized: now 1790707306531738 us; error 44 us; skew 500 ppm
Sep 29 18:41:46 dist-test-slave-j1xs krb5kdc[2724](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707306, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:41:46.537953 2737 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 18:41:46.538851 2737 webserver.cc:460] Webserver started at http://127.0.0.1:39319/ using document root <none> and password file <none>
I20260929 18:41:46.539445 2737 fs_manager.cc:359] Metadata directory not provided
I20260929 18:41:46.539537 2737 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:41:46.539774 2737 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:41:46.541811 2737 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790707036041078-13712-0/minicluster-data/master-0/data/instance:
uuid: "948c79fcb2994fc9bab5beae716cac3d"
format_stamp: "Formatted at 2026-09-29 18:41:46 on dist-test-slave-j1xs"
server_key: "200d626fd2a9b7e297a06c6c1245080e"
server_key_iv: "b5315d57b24b3f99eda7720a71f76321"
server_key_version: "encryptionkey@0"
I20260929 18:41:46.542443 2737 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790707036041078-13712-0/minicluster-data/master-0/wal/instance:
uuid: "948c79fcb2994fc9bab5beae716cac3d"
format_stamp: "Formatted at 2026-09-29 18:41:46 on dist-test-slave-j1xs"
server_key: "200d626fd2a9b7e297a06c6c1245080e"
server_key_iv: "b5315d57b24b3f99eda7720a71f76321"
server_key_version: "encryptionkey@0"
I20260929 18:41:46.546197 2737 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 18:41:46.548828 2752 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:41:46.550195 2737 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:41:46.550333 2737 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790707036041078-13712-0/minicluster-data/master-0/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790707036041078-13712-0/minicluster-data/master-0/wal
uuid: "948c79fcb2994fc9bab5beae716cac3d"
format_stamp: "Formatted at 2026-09-29 18:41:46 on dist-test-slave-j1xs"
server_key: "200d626fd2a9b7e297a06c6c1245080e"
server_key_iv: "b5315d57b24b3f99eda7720a71f76321"
server_key_version: "encryptionkey@0"
I20260929 18:41:46.550446 2737 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790707036041078-13712-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790707036041078-13712-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790707036041078-13712-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:41:46.554932 2737 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:41:46.557761 2737 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:41:46.557999 2737 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:41:46.564940 2737 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45163
I20260929 18:41:46.564960 2804 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45163 every 8 connection(s)
I20260929 18:41:46.565989 2737 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
I20260929 18:41:46.566502 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 2737
I20260929 18:41:46.566641 13712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790707036041078-13712-0/minicluster-data/master-0/wal/instance
I20260929 18:41:46.566975 13712 external_mini_cluster.cc:1428] Setting key 0a274845f8839dc8bd8a4646386f2224
I20260929 18:41:46.569092 2805 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:41:46.571995 2808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:46.568322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.43:34745 (local address 127.0.0.1:45163)
0929 18:41:46.568743 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0929 18:41:46.568757 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 18:41:46.568790 (+ 33us) server_negotiation.cc:407] Connection header received
0929 18:41:46.569476 (+ 686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:41:46.569493 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:41:46.569759 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:41:46.570140 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:41:46.570894 (+ 754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:41:46.570899 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:41:46.570910 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:41:46.570920 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:41:46.570968 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:41:46.570986 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:41:46.571138 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0929 18:41:46.571207 (+ 69us) server_negotiation.cc:299] Negotiation successful
0929 18:41:46.571360 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":119,"threads_started":1}
I20260929 18:41:46.574586 13712 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:41:46.574941 2805 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 948c79fcb2994fc9bab5beae716cac3d: Bootstrap starting.
I20260929 18:41:46.577221 2805 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 948c79fcb2994fc9bab5beae716cac3d: Neither blocks nor log segments found. Creating new log.
I20260929 18:41:46.578142 2805 log.cc:824] T 00000000000000000000000000000000 P 948c79fcb2994fc9bab5beae716cac3d: Log is configured to *not* fsync() on all Append() calls
I20260929 18:41:46.578238 2808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:46.576609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.43:35583 (local address 127.0.0.1:45163)
0929 18:41:46.576727 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0929 18:41:46.576731 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:41:46.576885 (+ 154us) server_negotiation.cc:407] Connection header received
0929 18:41:46.576977 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:41:46.576979 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:41:46.577012 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:41:46.577100 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:41:46.577779 (+ 679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:41:46.577783 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:41:46.577785 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:41:46.577796 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:41:46.577822 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:41:46.577834 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:41:46.577966 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0929 18:41:46.578055 (+ 89us) server_negotiation.cc:299] Negotiation successful
0929 18:41:46.578091 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 18:41:46.579916 2805 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 948c79fcb2994fc9bab5beae716cac3d: No bootstrap required, opened a new log
I20260929 18:41:46.583204 2805 raft_consensus.cc:348] T 00000000000000000000000000000000 P 948c79fcb2994fc9bab5beae716cac3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "948c79fcb2994fc9bab5beae716cac3d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45163 } }
I20260929 18:41:46.583442 2805 raft_consensus.cc:374] T 00000000000000000000000000000000 P 948c79fcb2994fc9bab5beae716cac3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:41:46.583501 2805 raft_consensus.cc:729] T 00000000000000000000000000000000 P 948c79fcb2994fc9bab5beae716cac3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 948c79fcb2994fc9bab5beae716cac3d, State: Initialized, Role: FOLLOWER
I20260929 18:41:46.583978 2805 consensus_queue.cc:260] T 00000000000000000000000000000000 P 948c79fcb2994fc9bab5beae716cac3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "948c79fcb2994fc9bab5beae716cac3d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45163 } }
I20260929 18:41:46.584141 2805 raft_consensus.cc:388] T 00000000000000000000000000000000 P 948c79fcb2994fc9bab5beae716cac3d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:41:46.584206 2805 raft_consensus.cc:482] T 00000000000000000000000000000000 P 948c79fcb2994fc9bab5beae716cac3d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:41:46.584298 2805 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 948c79fcb2994fc9bab5beae716cac3d [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:41:46.585108 2805 raft_consensus.cc:504] T 00000000000000000000000000000000 P 948c79fcb2994fc9bab5beae716cac3d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "948c79fcb2994fc9bab5beae716cac3d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45163 } }
I20260929 18:41:46.585456 2805 leader_election.cc:302] T 00000000000000000000000000000000 P 948c79fcb2994fc9bab5beae716cac3d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 948c79fcb2994fc9bab5beae716cac3d; no voters:
I20260929 18:41:46.585747 2805 leader_election.cc:288] T 00000000000000000000000000000000 P 948c79fcb2994fc9bab5beae716cac3d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:41:46.585826 2815 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 948c79fcb2994fc9bab5beae716cac3d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:41:46.586087 2815 raft_consensus.cc:686] T 00000000000000000000000000000000 P 948c79fcb2994fc9bab5beae716cac3d [term 1 LEADER]: Becoming Leader. State: Replica: 948c79fcb2994fc9bab5beae716cac3d, State: Running, Role: LEADER
I20260929 18:41:46.586395 2815 consensus_queue.cc:237] T 00000000000000000000000000000000 P 948c79fcb2994fc9bab5beae716cac3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "948c79fcb2994fc9bab5beae716cac3d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45163 } }
I20260929 18:41:46.586892 2805 sys_catalog.cc:564] T 00000000000000000000000000000000 P 948c79fcb2994fc9bab5beae716cac3d [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:41:46.588016 2816 sys_catalog.cc:455] T 00000000000000000000000000000000 P 948c79fcb2994fc9bab5beae716cac3d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "948c79fcb2994fc9bab5beae716cac3d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "948c79fcb2994fc9bab5beae716cac3d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45163 } } }
I20260929 18:41:46.588147 2816 sys_catalog.cc:458] T 00000000000000000000000000000000 P 948c79fcb2994fc9bab5beae716cac3d [sys.catalog]: This master's current role is: LEADER
I20260929 18:41:46.588410 2817 sys_catalog.cc:455] T 00000000000000000000000000000000 P 948c79fcb2994fc9bab5beae716cac3d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 948c79fcb2994fc9bab5beae716cac3d. Latest consensus state: current_term: 1 leader_uuid: "948c79fcb2994fc9bab5beae716cac3d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "948c79fcb2994fc9bab5beae716cac3d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45163 } } }
I20260929 18:41:46.588516 2817 sys_catalog.cc:458] T 00000000000000000000000000000000 P 948c79fcb2994fc9bab5beae716cac3d [sys.catalog]: This master's current role is: LEADER
I20260929 18:41:46.588537 2824 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:41:46.591563 2824 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:41:46.596844 2824 catalog_manager.cc:1347] Generated new cluster ID: b993c8a72e33482692038099aea1e785
I20260929 18:41:46.596920 2824 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:41:46.614735 2824 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:41:46.615072 2824 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:41:46.625306 2824 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 948c79fcb2994fc9bab5beae716cac3d: Generated new TSK 0
I20260929 18:41:46.626194 2824 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:41:46.634789 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 2737
2026-09-29T18:41:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1388 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790707036041078-13712-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:41:46 dist-test-slave-j1xs krb5kdc[2836](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:41:46 dist-test-slave-j1xs krb5kdc[2836](info): set up 2 sockets
Sep 29 18:41:46 dist-test-slave-j1xs krb5kdc[2836](info): commencing operation
krb5kdc: starting...
W20260929 18:41:47.674289 13712 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790707036041078-13712-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790707036041078-13712-0/krb5kdc/test-user.keytab.
Sep 29 18:41:47 dist-test-slave-j1xs krb5kdc[2836](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707307, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:41:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:41:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.100.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.100.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.100.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.62.keytab.
Entry for principal kudu/127.13.100.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.62.keytab.
I20260929 18:41:47.797063 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790707036041078-13712-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790707036041078-13712-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790707036041078-13712-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.100.62:40963
--webserver_interface=127.13.100.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.62.keytab
--principal=kudu/127.13.100.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.100.20:41599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.100.62:40963
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790707036041078-13712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790707036041078-13712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790707036041078-13712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:41:47.907969 2849 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:41:47.908279 2849 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:41:47.908358 2849 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:41:47.912395 2849 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:41:47.912490 2849 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:41:47.912529 2849 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:41:47.912586 2849 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:41:47.912626 2849 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:41:47.918131 2849 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.100.20:41599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790707036041078-13712-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790707036041078-13712-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.100.62:40963
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.62.keytab
--principal=kudu/127.13.100.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.100.62:40963
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.100.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2849
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790707036041078-13712-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:41:47.919216 2849 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:41:47.920341 2849 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 18:41:47.928813 2849 server_base.cc:1034] running on GCE node
W20260929 18:41:47.928818 2854 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:41:47.928856 2857 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:41:47.928799 2855 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:41:47.929570 2849 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:41:47.930689 2849 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:41:47.931871 2849 hybrid_clock.cc:648] HybridClock initialized: now 1790707307931858 us; error 34 us; skew 500 ppm
Sep 29 18:41:47 dist-test-slave-j1xs krb5kdc[2836](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707307, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:41:47.934554 2849 init.cc:377] Logged in from keytab as kudu/127.13.100.62@KRBTEST.COM (short username kudu)
I20260929 18:41:47.935425 2849 webserver.cc:460] Webserver started at http://127.13.100.62:38667/ using document root <none> and password file <none>
I20260929 18:41:47.935963 2849 fs_manager.cc:359] Metadata directory not provided
I20260929 18:41:47.936051 2849 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:41:47.936268 2849 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:41:47.938575 2849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790707036041078-13712-0/minicluster-data/master-0/data/instance:
uuid: "f12902d7cbbc4f249cbf2854e68384ca"
format_stamp: "Formatted at 2026-09-29 18:41:47 on dist-test-slave-j1xs"
server_key: "a71f336b3e81c348d0afe01e510f9a67"
server_key_iv: "9db2a5f2fdc3c0ae2ed8b442cf33d7b0"
server_key_version: "encryptionkey@0"
I20260929 18:41:47.939121 2849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790707036041078-13712-0/minicluster-data/master-0/wal/instance:
uuid: "f12902d7cbbc4f249cbf2854e68384ca"
format_stamp: "Formatted at 2026-09-29 18:41:47 on dist-test-slave-j1xs"
server_key: "a71f336b3e81c348d0afe01e510f9a67"
server_key_iv: "9db2a5f2fdc3c0ae2ed8b442cf33d7b0"
server_key_version: "encryptionkey@0"
I20260929 18:41:47.942734 2849 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 18:41:47.945044 2864 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:41:47.946390 2849 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:41:47.946520 2849 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790707036041078-13712-0/minicluster-data/master-0/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790707036041078-13712-0/minicluster-data/master-0/wal
uuid: "f12902d7cbbc4f249cbf2854e68384ca"
format_stamp: "Formatted at 2026-09-29 18:41:47 on dist-test-slave-j1xs"
server_key: "a71f336b3e81c348d0afe01e510f9a67"
server_key_iv: "9db2a5f2fdc3c0ae2ed8b442cf33d7b0"
server_key_version: "encryptionkey@0"
I20260929 18:41:47.946622 2849 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790707036041078-13712-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790707036041078-13712-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790707036041078-13712-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:41:47.955894 2849 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:41:47.959193 2849 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:41:47.959393 2849 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:41:47.966248 2849 rpc_server.cc:307] RPC server started. Bound to: 127.13.100.62:40963
I20260929 18:41:47.966269 2916 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.100.62:40963 every 8 connection(s)
I20260929 18:41:47.967151 2849 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
I20260929 18:41:47.969841 2917 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:41:47.973316 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 2849
I20260929 18:41:47.973476 13712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790707036041078-13712-0/minicluster-data/master-0/wal/instance
I20260929 18:41:47.973732 13712 external_mini_cluster.cc:1428] Setting key 8d35194114abe962fa85ca347b25b04d
I20260929 18:41:47.975770 2917 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f12902d7cbbc4f249cbf2854e68384ca: Bootstrap starting.
I20260929 18:41:47.978564 2917 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f12902d7cbbc4f249cbf2854e68384ca: Neither blocks nor log segments found. Creating new log.
I20260929 18:41:47.979446 2917 log.cc:824] T 00000000000000000000000000000000 P f12902d7cbbc4f249cbf2854e68384ca: Log is configured to *not* fsync() on all Append() calls
I20260929 18:41:47.981678 2917 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f12902d7cbbc4f249cbf2854e68384ca: No bootstrap required, opened a new log
Sep 29 18:41:47 dist-test-slave-j1xs krb5kdc[2836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707307, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.100.62@KRBTEST.COM
I20260929 18:41:47.985790 2917 raft_consensus.cc:348] T 00000000000000000000000000000000 P f12902d7cbbc4f249cbf2854e68384ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f12902d7cbbc4f249cbf2854e68384ca" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 40963 } }
I20260929 18:41:47.986099 2917 raft_consensus.cc:374] T 00000000000000000000000000000000 P f12902d7cbbc4f249cbf2854e68384ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:41:47.986169 2917 raft_consensus.cc:729] T 00000000000000000000000000000000 P f12902d7cbbc4f249cbf2854e68384ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f12902d7cbbc4f249cbf2854e68384ca, State: Initialized, Role: FOLLOWER
I20260929 18:41:47.986653 2917 consensus_queue.cc:260] T 00000000000000000000000000000000 P f12902d7cbbc4f249cbf2854e68384ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f12902d7cbbc4f249cbf2854e68384ca" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 40963 } }
I20260929 18:41:47.986891 2917 raft_consensus.cc:388] T 00000000000000000000000000000000 P f12902d7cbbc4f249cbf2854e68384ca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:41:47.986991 2917 raft_consensus.cc:482] T 00000000000000000000000000000000 P f12902d7cbbc4f249cbf2854e68384ca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:41:47.987088 2917 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f12902d7cbbc4f249cbf2854e68384ca [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:41:47.987975 2917 raft_consensus.cc:504] T 00000000000000000000000000000000 P f12902d7cbbc4f249cbf2854e68384ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f12902d7cbbc4f249cbf2854e68384ca" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 40963 } }
I20260929 18:41:47.988302 2917 leader_election.cc:302] T 00000000000000000000000000000000 P f12902d7cbbc4f249cbf2854e68384ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f12902d7cbbc4f249cbf2854e68384ca; no voters:
I20260929 18:41:47.988616 2917 leader_election.cc:288] T 00000000000000000000000000000000 P f12902d7cbbc4f249cbf2854e68384ca [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:41:47.988685 2922 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f12902d7cbbc4f249cbf2854e68384ca [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:41:47.988888 2922 raft_consensus.cc:686] T 00000000000000000000000000000000 P f12902d7cbbc4f249cbf2854e68384ca [term 1 LEADER]: Becoming Leader. State: Replica: f12902d7cbbc4f249cbf2854e68384ca, State: Running, Role: LEADER
I20260929 18:41:47.989065 2920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:47.975117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60165 (local address 127.13.100.62:40963)
0929 18:41:47.975519 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0929 18:41:47.975539 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 18:41:47.975582 (+ 43us) server_negotiation.cc:407] Connection header received
0929 18:41:47.976424 (+ 842us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:41:47.976446 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:41:47.976849 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:41:47.977229 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:41:47.977901 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:47.979419 (+ 1518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:47.980516 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:47.981193 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:47.981358 (+ 165us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:47.983765 (+ 2407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:41:47.983771 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:41:47.983786 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:41:47.983818 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:41:47.985749 (+ 1931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:41:47.986391 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:41:47.986397 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:41:47.986402 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:41:47.986538 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:41:47.986856 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:41:47.986860 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:41:47.986862 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:41:47.987335 (+ 473us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:41:47.987578 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0929 18:41:47.987871 (+ 293us) server_negotiation.cc:299] Negotiation successful
0929 18:41:47.988121 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":98,"threads_started":1}
I20260929 18:41:47.989243 2922 consensus_queue.cc:237] T 00000000000000000000000000000000 P f12902d7cbbc4f249cbf2854e68384ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f12902d7cbbc4f249cbf2854e68384ca" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 40963 } }
I20260929 18:41:47.989862 2917 sys_catalog.cc:564] T 00000000000000000000000000000000 P f12902d7cbbc4f249cbf2854e68384ca [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:41:47.990861 2923 sys_catalog.cc:455] T 00000000000000000000000000000000 P f12902d7cbbc4f249cbf2854e68384ca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f12902d7cbbc4f249cbf2854e68384ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f12902d7cbbc4f249cbf2854e68384ca" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 40963 } } }
I20260929 18:41:47.991573 2923 sys_catalog.cc:458] T 00000000000000000000000000000000 P f12902d7cbbc4f249cbf2854e68384ca [sys.catalog]: This master's current role is: LEADER
I20260929 18:41:47.991467 2924 sys_catalog.cc:455] T 00000000000000000000000000000000 P f12902d7cbbc4f249cbf2854e68384ca [sys.catalog]: SysCatalogTable state changed. Reason: New leader f12902d7cbbc4f249cbf2854e68384ca. Latest consensus state: current_term: 1 leader_uuid: "f12902d7cbbc4f249cbf2854e68384ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f12902d7cbbc4f249cbf2854e68384ca" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 40963 } } }
I20260929 18:41:47.991899 2924 sys_catalog.cc:458] T 00000000000000000000000000000000 P f12902d7cbbc4f249cbf2854e68384ca [sys.catalog]: This master's current role is: LEADER
I20260929 18:41:47.992493 13712 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260929 18:41:47.994681 2937 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f12902d7cbbc4f249cbf2854e68384ca: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:41:47.994765 2937 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:41:47.994843 2936 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:41:47.995821 2936 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:41:48.001986 2936 catalog_manager.cc:1347] Generated new cluster ID: 4aadfb7bcefc4def8a8322846e81f0b1
I20260929 18:41:48.002110 2936 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:41:48.003700 2920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:47.994435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49553 (local address 127.13.100.62:40963)
0929 18:41:47.994538 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0929 18:41:47.994542 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:41:47.995089 (+ 547us) server_negotiation.cc:407] Connection header received
0929 18:41:47.995215 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:41:47.995218 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:41:47.995266 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:41:47.995344 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:41:47.996143 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:47.996918 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:47.997929 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:47.998163 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:47.998278 (+ 115us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:47.999963 (+ 1685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:41:47.999966 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:41:47.999968 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:41:47.999988 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:41:48.001693 (+ 1705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:41:48.002341 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:41:48.002344 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:41:48.002346 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:41:48.002426 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:41:48.002794 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:41:48.002797 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:41:48.002799 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:41:48.003034 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:41:48.003160 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0929 18:41:48.003448 (+ 288us) server_negotiation.cc:299] Negotiation successful
0929 18:41:48.003537 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":38}
I20260929 18:41:48.011541 2936 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:41:48.012635 2936 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:41:48.035764 2936 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f12902d7cbbc4f249cbf2854e68384ca: Generated new TSK 0
I20260929 18:41:48.036544 2936 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:41:48.046269 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 2849
W20260929 18:41:48.051509 2834 connection.cc:535] client connection to 127.13.100.62:40963 recv error: Network error: failed to read from TLS socket (remote: 127.13.100.62:40963): Connection reset by peer (error 104)
2026-09-29T18:41:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1411 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790707036041078-13712-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:41:48 dist-test-slave-j1xs krb5kdc[2948](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:41:48 dist-test-slave-j1xs krb5kdc[2948](info): set up 2 sockets
Sep 29 18:41:48 dist-test-slave-j1xs krb5kdc[2948](info): commencing operation
krb5kdc: starting...
W20260929 18:41:49.082172 13712 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790707036041078-13712-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790707036041078-13712-0/krb5kdc/test-user.keytab.
Sep 29 18:41:49 dist-test-slave-j1xs krb5kdc[2948](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707309, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:41:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:41:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.100.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.100.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.100.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.62.keytab.
Entry for principal kudu/127.13.100.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.62.keytab.
I20260929 18:41:49.205376 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790707036041078-13712-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790707036041078-13712-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790707036041078-13712-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.100.62:39999
--webserver_interface=127.13.100.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.62.keytab
--principal=kudu/127.13.100.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.100.20:44561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.100.62:39999
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790707036041078-13712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790707036041078-13712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790707036041078-13712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:41:49.314029 2961 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:41:49.314327 2961 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:41:49.314414 2961 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:41:49.318420 2961 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:41:49.318516 2961 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:41:49.318554 2961 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:41:49.318612 2961 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:41:49.318651 2961 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:41:49.324263 2961 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.100.20:44561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790707036041078-13712-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790707036041078-13712-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.100.62:39999
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.62.keytab
--principal=kudu/127.13.100.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.100.62:39999
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.100.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2961
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790707036041078-13712-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:41:49.325351 2961 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:41:49.326378 2961 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:41:49.334575 2966 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:41:49.334530 2961 server_base.cc:1034] running on GCE node
W20260929 18:41:49.334590 2967 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:41:49.334515 2969 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:41:49.335315 2961 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:41:49.336397 2961 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:41:49.337584 2961 hybrid_clock.cc:648] HybridClock initialized: now 1790707309337572 us; error 24 us; skew 500 ppm
Sep 29 18:41:49 dist-test-slave-j1xs krb5kdc[2948](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707309, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:41:49.340598 2961 init.cc:377] Logged in from keytab as kudu/127.13.100.62@KRBTEST.COM (short username kudu)
I20260929 18:41:49.341511 2961 webserver.cc:460] Webserver started at http://127.13.100.62:38215/ using document root <none> and password file <none>
I20260929 18:41:49.342123 2961 fs_manager.cc:359] Metadata directory not provided
I20260929 18:41:49.342211 2961 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:41:49.342447 2961 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:41:49.344544 2961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790707036041078-13712-0/minicluster-data/master-0/data/instance:
uuid: "3b40883456644966b171d9ea16da42fb"
format_stamp: "Formatted at 2026-09-29 18:41:49 on dist-test-slave-j1xs"
server_key: "531b9444d33e3ce46ddc441ea11a14be"
server_key_iv: "193ccb1b15d6240ee00a97312ebc320c"
server_key_version: "encryptionkey@0"
I20260929 18:41:49.345096 2961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790707036041078-13712-0/minicluster-data/master-0/wal/instance:
uuid: "3b40883456644966b171d9ea16da42fb"
format_stamp: "Formatted at 2026-09-29 18:41:49 on dist-test-slave-j1xs"
server_key: "531b9444d33e3ce46ddc441ea11a14be"
server_key_iv: "193ccb1b15d6240ee00a97312ebc320c"
server_key_version: "encryptionkey@0"
I20260929 18:41:49.348863 2961 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 18:41:49.351347 2976 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:41:49.352725 2961 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 18:41:49.352895 2961 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790707036041078-13712-0/minicluster-data/master-0/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790707036041078-13712-0/minicluster-data/master-0/wal
uuid: "3b40883456644966b171d9ea16da42fb"
format_stamp: "Formatted at 2026-09-29 18:41:49 on dist-test-slave-j1xs"
server_key: "531b9444d33e3ce46ddc441ea11a14be"
server_key_iv: "193ccb1b15d6240ee00a97312ebc320c"
server_key_version: "encryptionkey@0"
I20260929 18:41:49.353024 2961 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790707036041078-13712-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790707036041078-13712-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790707036041078-13712-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:41:49.370327 2961 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:41:49.373508 2961 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:41:49.373723 2961 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:41:49.381071 2961 rpc_server.cc:307] RPC server started. Bound to: 127.13.100.62:39999
I20260929 18:41:49.381098 3028 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.100.62:39999 every 8 connection(s)
I20260929 18:41:49.382052 2961 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
I20260929 18:41:49.384718 3029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:41:49.390386 3029 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b40883456644966b171d9ea16da42fb: Bootstrap starting.
I20260929 18:41:49.391587 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 2961
I20260929 18:41:49.391731 13712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790707036041078-13712-0/minicluster-data/master-0/wal/instance
I20260929 18:41:49.392030 13712 external_mini_cluster.cc:1428] Setting key 7931be6ef91416ce47f66e348b303e94
I20260929 18:41:49.392994 3029 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3b40883456644966b171d9ea16da42fb: Neither blocks nor log segments found. Creating new log.
I20260929 18:41:49.393980 3029 log.cc:824] T 00000000000000000000000000000000 P 3b40883456644966b171d9ea16da42fb: Log is configured to *not* fsync() on all Append() calls
I20260929 18:41:49.396534 3029 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b40883456644966b171d9ea16da42fb: No bootstrap required, opened a new log
I20260929 18:41:49.400754 3029 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3b40883456644966b171d9ea16da42fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b40883456644966b171d9ea16da42fb" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 39999 } }
I20260929 18:41:49.401067 3029 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3b40883456644966b171d9ea16da42fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:41:49.401139 3029 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3b40883456644966b171d9ea16da42fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b40883456644966b171d9ea16da42fb, State: Initialized, Role: FOLLOWER
I20260929 18:41:49.401558 3029 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3b40883456644966b171d9ea16da42fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b40883456644966b171d9ea16da42fb" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 39999 } }
I20260929 18:41:49.401706 3029 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3b40883456644966b171d9ea16da42fb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:41:49.401774 3029 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3b40883456644966b171d9ea16da42fb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:41:49.401870 3029 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3b40883456644966b171d9ea16da42fb [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:41:49.401937 3032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:49.393249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43189 (local address 127.13.100.62:39999)
0929 18:41:49.393784 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0929 18:41:49.393805 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0929 18:41:49.393852 (+ 47us) server_negotiation.cc:407] Connection header received
0929 18:41:49.394806 (+ 954us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:41:49.394825 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:41:49.395200 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:41:49.395604 (+ 404us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:41:49.396023 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:49.397219 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:49.398494 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:49.398965 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:49.399095 (+ 130us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:49.400064 (+ 969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:41:49.400069 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:41:49.400081 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:41:49.400477 (+ 396us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:41:49.400527 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:41:49.400549 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:41:49.400658 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 18:41:49.400940 (+ 282us) server_negotiation.cc:299] Negotiation successful
0929 18:41:49.401081 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":116,"threads_started":1}
I20260929 18:41:49.402762 3029 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3b40883456644966b171d9ea16da42fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b40883456644966b171d9ea16da42fb" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 39999 } }
I20260929 18:41:49.403120 3029 leader_election.cc:302] T 00000000000000000000000000000000 P 3b40883456644966b171d9ea16da42fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3b40883456644966b171d9ea16da42fb; no voters:
I20260929 18:41:49.403451 3029 leader_election.cc:288] T 00000000000000000000000000000000 P 3b40883456644966b171d9ea16da42fb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:41:49.403582 3034 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3b40883456644966b171d9ea16da42fb [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:41:49.403795 3034 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3b40883456644966b171d9ea16da42fb [term 1 LEADER]: Becoming Leader. State: Replica: 3b40883456644966b171d9ea16da42fb, State: Running, Role: LEADER
I20260929 18:41:49.404137 3034 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3b40883456644966b171d9ea16da42fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b40883456644966b171d9ea16da42fb" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 39999 } }
I20260929 18:41:49.404695 3029 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3b40883456644966b171d9ea16da42fb [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:41:49.404983 13712 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:41:49.405339 3036 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b40883456644966b171d9ea16da42fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3b40883456644966b171d9ea16da42fb. Latest consensus state: current_term: 1 leader_uuid: "3b40883456644966b171d9ea16da42fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b40883456644966b171d9ea16da42fb" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 39999 } } }
I20260929 18:41:49.406082 3036 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b40883456644966b171d9ea16da42fb [sys.catalog]: This master's current role is: LEADER
I20260929 18:41:49.405962 3035 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b40883456644966b171d9ea16da42fb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3b40883456644966b171d9ea16da42fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b40883456644966b171d9ea16da42fb" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 39999 } } }
I20260929 18:41:49.406339 3035 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b40883456644966b171d9ea16da42fb [sys.catalog]: This master's current role is: LEADER
I20260929 18:41:49.407013 3044 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:41:49.409909 3044 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:41:49.412141 3032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:49.407326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35019 (local address 127.13.100.62:39999)
0929 18:41:49.407433 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0929 18:41:49.407438 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:41:49.407925 (+ 487us) server_negotiation.cc:407] Connection header received
0929 18:41:49.408025 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:41:49.408028 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:41:49.408070 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:41:49.408158 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:41:49.408441 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:49.409183 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:49.410260 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:49.410601 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:49.410712 (+ 111us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:49.411629 (+ 917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:41:49.411633 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:41:49.411636 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:41:49.411648 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:41:49.411676 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:41:49.411689 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:41:49.411777 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0929 18:41:49.411968 (+ 191us) server_negotiation.cc:299] Negotiation successful
0929 18:41:49.412012 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260929 18:41:49.415184 3044 catalog_manager.cc:1347] Generated new cluster ID: 7fef45c375ba459ab5d662c198f03b7c
I20260929 18:41:49.415241 3044 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:41:49.436798 3044 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:41:49.438128 3044 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:41:49.451257 3044 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3b40883456644966b171d9ea16da42fb: Generated new TSK 0
I20260929 18:41:49.452129 3044 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:41:49.458436 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 2961
W20260929 18:41:49.464071 2946 connection.cc:535] client connection to 127.13.100.62:39999 recv error: Network error: failed to read from TLS socket (remote: 127.13.100.62:39999): Connection reset by peer (error 104)
2026-09-29T18:41:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1412 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790707036041078-13712-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:41:49 dist-test-slave-j1xs krb5kdc[3060](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:41:49 dist-test-slave-j1xs krb5kdc[3060](info): set up 2 sockets
Sep 29 18:41:49 dist-test-slave-j1xs krb5kdc[3060](info): commencing operation
krb5kdc: starting...
W20260929 18:41:50.514102 13712 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790707036041078-13712-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790707036041078-13712-0/krb5kdc/test-user.keytab.
Sep 29 18:41:50 dist-test-slave-j1xs krb5kdc[3060](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707310, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:41:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:41:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.100.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.100.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.100.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.62.keytab.
Entry for principal kudu/127.13.100.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.62.keytab.
I20260929 18:41:50.640916 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790707036041078-13712-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790707036041078-13712-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790707036041078-13712-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.100.62:36039
--webserver_interface=127.13.100.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.62.keytab
--principal=kudu/127.13.100.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.100.20:43671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.100.62:36039
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790707036041078-13712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790707036041078-13712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790707036041078-13712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:41:50.752794 3073 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:41:50.753123 3073 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:41:50.753202 3073 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:41:50.757376 3073 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:41:50.757476 3073 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:41:50.757521 3073 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:41:50.757583 3073 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:41:50.757622 3073 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:41:50.763199 3073 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.100.20:43671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790707036041078-13712-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790707036041078-13712-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.100.62:36039
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.62.keytab
--principal=kudu/127.13.100.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.100.62:36039
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.100.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3073
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790707036041078-13712-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:41:50.764317 3073 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:41:50.765333 3073 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:41:50.773957 3081 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:41:50.773950 3078 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:41:50.773950 3079 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:41:50.774186 3073 server_base.cc:1034] running on GCE node
I20260929 18:41:50.774892 3073 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:41:50.775993 3073 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:41:50.777177 3073 hybrid_clock.cc:648] HybridClock initialized: now 1790707310777159 us; error 38 us; skew 500 ppm
Sep 29 18:41:50 dist-test-slave-j1xs krb5kdc[3060](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707310, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:41:50.779968 3073 init.cc:377] Logged in from keytab as kudu/127.13.100.62@KRBTEST.COM (short username kudu)
I20260929 18:41:50.780850 3073 webserver.cc:460] Webserver started at http://127.13.100.62:35935/ using document root <none> and password file <none>
I20260929 18:41:50.781365 3073 fs_manager.cc:359] Metadata directory not provided
I20260929 18:41:50.781423 3073 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:41:50.781579 3073 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:41:50.783633 3073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790707036041078-13712-0/minicluster-data/master-0/data/instance:
uuid: "d068f3aa08e84803a4e1e6677743fb0d"
format_stamp: "Formatted at 2026-09-29 18:41:50 on dist-test-slave-j1xs"
server_key: "5d03c9787b97fb26e818a0cd27a178ad"
server_key_iv: "952b8e1d6e92cdd174db8fc38ecbe4e3"
server_key_version: "encryptionkey@0"
I20260929 18:41:50.784145 3073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790707036041078-13712-0/minicluster-data/master-0/wal/instance:
uuid: "d068f3aa08e84803a4e1e6677743fb0d"
format_stamp: "Formatted at 2026-09-29 18:41:50 on dist-test-slave-j1xs"
server_key: "5d03c9787b97fb26e818a0cd27a178ad"
server_key_iv: "952b8e1d6e92cdd174db8fc38ecbe4e3"
server_key_version: "encryptionkey@0"
I20260929 18:41:50.787875 3073 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 18:41:50.790478 3088 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:41:50.791759 3073 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:41:50.791858 3073 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790707036041078-13712-0/minicluster-data/master-0/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790707036041078-13712-0/minicluster-data/master-0/wal
uuid: "d068f3aa08e84803a4e1e6677743fb0d"
format_stamp: "Formatted at 2026-09-29 18:41:50 on dist-test-slave-j1xs"
server_key: "5d03c9787b97fb26e818a0cd27a178ad"
server_key_iv: "952b8e1d6e92cdd174db8fc38ecbe4e3"
server_key_version: "encryptionkey@0"
I20260929 18:41:50.792023 3073 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790707036041078-13712-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790707036041078-13712-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790707036041078-13712-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:41:50.796837 3073 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:41:50.800393 3073 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:41:50.800626 3073 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:41:50.808085 3073 rpc_server.cc:307] RPC server started. Bound to: 127.13.100.62:36039
I20260929 18:41:50.808454 3140 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.100.62:36039 every 8 connection(s)
I20260929 18:41:50.809113 3073 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
I20260929 18:41:50.812038 3141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:41:50.817116 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 3073
I20260929 18:41:50.817260 13712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790707036041078-13712-0/minicluster-data/master-0/wal/instance
I20260929 18:41:50.817539 13712 external_mini_cluster.cc:1428] Setting key 7729e35251bdd10cc2328ae70d8b5287
I20260929 18:41:50.817981 3141 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d068f3aa08e84803a4e1e6677743fb0d: Bootstrap starting.
I20260929 18:41:50.820611 3141 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d068f3aa08e84803a4e1e6677743fb0d: Neither blocks nor log segments found. Creating new log.
I20260929 18:41:50.821508 3141 log.cc:824] T 00000000000000000000000000000000 P d068f3aa08e84803a4e1e6677743fb0d: Log is configured to *not* fsync() on all Append() calls
I20260929 18:41:50.823383 3144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:50.819044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33493 (local address 127.13.100.62:36039)
0929 18:41:50.819479 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0929 18:41:50.819514 (+ 35us) server_negotiation.cc:399] Waiting for connection header
0929 18:41:50.819556 (+ 42us) server_negotiation.cc:407] Connection header received
0929 18:41:50.820367 (+ 811us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:41:50.820389 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:41:50.820777 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:41:50.821174 (+ 397us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:41:50.822219 (+ 1045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:41:50.822224 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:41:50.822244 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:41:50.822254 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:41:50.822307 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:41:50.822326 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:41:50.822446 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0929 18:41:50.822559 (+ 113us) server_negotiation.cc:299] Negotiation successful
0929 18:41:50.822711 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":114,"threads_started":1}
I20260929 18:41:50.823913 3141 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d068f3aa08e84803a4e1e6677743fb0d: No bootstrap required, opened a new log
I20260929 18:41:50.826165 13712 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:41:50.828063 3141 raft_consensus.cc:348] T 00000000000000000000000000000000 P d068f3aa08e84803a4e1e6677743fb0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d068f3aa08e84803a4e1e6677743fb0d" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 36039 } }
I20260929 18:41:50.828341 3141 raft_consensus.cc:374] T 00000000000000000000000000000000 P d068f3aa08e84803a4e1e6677743fb0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:41:50.828409 3141 raft_consensus.cc:729] T 00000000000000000000000000000000 P d068f3aa08e84803a4e1e6677743fb0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d068f3aa08e84803a4e1e6677743fb0d, State: Initialized, Role: FOLLOWER
I20260929 18:41:50.828884 3141 consensus_queue.cc:260] T 00000000000000000000000000000000 P d068f3aa08e84803a4e1e6677743fb0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d068f3aa08e84803a4e1e6677743fb0d" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 36039 } }
I20260929 18:41:50.829056 3141 raft_consensus.cc:388] T 00000000000000000000000000000000 P d068f3aa08e84803a4e1e6677743fb0d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:41:50.829128 3141 raft_consensus.cc:482] T 00000000000000000000000000000000 P d068f3aa08e84803a4e1e6677743fb0d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:41:50.829223 3141 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d068f3aa08e84803a4e1e6677743fb0d [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:41:50.830088 3144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:50.828171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52053 (local address 127.13.100.62:36039)
0929 18:41:50.828293 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0929 18:41:50.828297 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:41:50.828475 (+ 178us) server_negotiation.cc:407] Connection header received
0929 18:41:50.828565 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:41:50.828568 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:41:50.828603 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:41:50.828684 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:41:50.829621 (+ 937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:41:50.829623 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:41:50.829625 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:41:50.829635 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:41:50.829659 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:41:50.829669 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:41:50.829741 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 18:41:50.829841 (+ 100us) server_negotiation.cc:299] Negotiation successful
0929 18:41:50.829870 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 18:41:50.830922 3141 raft_consensus.cc:504] T 00000000000000000000000000000000 P d068f3aa08e84803a4e1e6677743fb0d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d068f3aa08e84803a4e1e6677743fb0d" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 36039 } }
I20260929 18:41:50.831298 3141 leader_election.cc:302] T 00000000000000000000000000000000 P d068f3aa08e84803a4e1e6677743fb0d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d068f3aa08e84803a4e1e6677743fb0d; no voters:
I20260929 18:41:50.831607 3141 leader_election.cc:288] T 00000000000000000000000000000000 P d068f3aa08e84803a4e1e6677743fb0d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:41:50.831776 3151 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d068f3aa08e84803a4e1e6677743fb0d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:41:50.832034 3151 raft_consensus.cc:686] T 00000000000000000000000000000000 P d068f3aa08e84803a4e1e6677743fb0d [term 1 LEADER]: Becoming Leader. State: Replica: d068f3aa08e84803a4e1e6677743fb0d, State: Running, Role: LEADER
I20260929 18:41:50.832393 3151 consensus_queue.cc:237] T 00000000000000000000000000000000 P d068f3aa08e84803a4e1e6677743fb0d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d068f3aa08e84803a4e1e6677743fb0d" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 36039 } }
I20260929 18:41:50.832886 3141 sys_catalog.cc:564] T 00000000000000000000000000000000 P d068f3aa08e84803a4e1e6677743fb0d [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:41:50.834069 3152 sys_catalog.cc:455] T 00000000000000000000000000000000 P d068f3aa08e84803a4e1e6677743fb0d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d068f3aa08e84803a4e1e6677743fb0d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d068f3aa08e84803a4e1e6677743fb0d" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 36039 } } }
I20260929 18:41:50.834203 3152 sys_catalog.cc:458] T 00000000000000000000000000000000 P d068f3aa08e84803a4e1e6677743fb0d [sys.catalog]: This master's current role is: LEADER
I20260929 18:41:50.834520 3153 sys_catalog.cc:455] T 00000000000000000000000000000000 P d068f3aa08e84803a4e1e6677743fb0d [sys.catalog]: SysCatalogTable state changed. Reason: New leader d068f3aa08e84803a4e1e6677743fb0d. Latest consensus state: current_term: 1 leader_uuid: "d068f3aa08e84803a4e1e6677743fb0d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d068f3aa08e84803a4e1e6677743fb0d" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 36039 } } }
I20260929 18:41:50.834630 3160 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:41:50.834642 3153 sys_catalog.cc:458] T 00000000000000000000000000000000 P d068f3aa08e84803a4e1e6677743fb0d [sys.catalog]: This master's current role is: LEADER
I20260929 18:41:50.837985 3160 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:41:50.843969 3160 catalog_manager.cc:1347] Generated new cluster ID: f8a806b5139d4db9abfd694eaf0a0acc
I20260929 18:41:50.844099 3160 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:41:50.864002 3160 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:41:50.864328 3160 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:41:50.880046 3160 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d068f3aa08e84803a4e1e6677743fb0d: Generated new TSK 0
I20260929 18:41:50.880764 3160 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:41:50.888302 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 3073
2026-09-29T18:41:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1429 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790707036041078-13712-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:41:50 dist-test-slave-j1xs krb5kdc[3172](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:41:50 dist-test-slave-j1xs krb5kdc[3172](info): set up 2 sockets
Sep 29 18:41:50 dist-test-slave-j1xs krb5kdc[3172](info): commencing operation
krb5kdc: starting...
W20260929 18:41:51.918169 13712 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790707036041078-13712-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790707036041078-13712-0/krb5kdc/test-user.keytab.
Sep 29 18:41:52 dist-test-slave-j1xs krb5kdc[3172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707312, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:41:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:41:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.100.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.100.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.100.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.62.keytab.
Entry for principal kudu/127.13.100.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.62.keytab.
I20260929 18:41:52.041821 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790707036041078-13712-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790707036041078-13712-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790707036041078-13712-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.100.62:37687
--webserver_interface=127.13.100.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.62.keytab
--principal=kudu/127.13.100.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.100.20:39401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.100.62:37687
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790707036041078-13712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790707036041078-13712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790707036041078-13712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:41:52.153831 3185 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:41:52.154199 3185 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:41:52.154301 3185 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:41:52.158365 3185 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:41:52.158444 3185 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:41:52.158481 3185 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:41:52.158540 3185 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:41:52.158581 3185 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:41:52.163852 3185 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.100.20:39401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790707036041078-13712-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790707036041078-13712-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.100.62:37687
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.62.keytab
--principal=kudu/127.13.100.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.100.62:37687
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.100.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3185
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790707036041078-13712-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:41:52.164867 3185 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:41:52.165827 3185 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 18:41:52.174037 3185 server_base.cc:1034] running on GCE node
W20260929 18:41:52.173992 3193 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:41:52.173992 3191 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:41:52.173992 3190 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:41:52.174829 3185 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:41:52.175859 3185 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:41:52.177083 3185 hybrid_clock.cc:648] HybridClock initialized: now 1790707312177053 us; error 42 us; skew 500 ppm
Sep 29 18:41:52 dist-test-slave-j1xs krb5kdc[3172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707312, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:41:52.183360 3185 init.cc:377] Logged in from keytab as kudu/127.13.100.62@KRBTEST.COM (short username kudu)
I20260929 18:41:52.184288 3185 webserver.cc:460] Webserver started at http://127.13.100.62:33239/ using document root <none> and password file <none>
I20260929 18:41:52.184828 3185 fs_manager.cc:359] Metadata directory not provided
I20260929 18:41:52.184887 3185 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:41:52.185113 3185 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:41:52.187103 3185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790707036041078-13712-0/minicluster-data/master-0/data/instance:
uuid: "43f00870ee9d49b98c538ccf6229e9c3"
format_stamp: "Formatted at 2026-09-29 18:41:52 on dist-test-slave-j1xs"
server_key: "20b61cfa3309a596ea8248119158022a"
server_key_iv: "09a3b7b4ec020f41bff5064013a0ce75"
server_key_version: "encryptionkey@0"
I20260929 18:41:52.187637 3185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790707036041078-13712-0/minicluster-data/master-0/wal/instance:
uuid: "43f00870ee9d49b98c538ccf6229e9c3"
format_stamp: "Formatted at 2026-09-29 18:41:52 on dist-test-slave-j1xs"
server_key: "20b61cfa3309a596ea8248119158022a"
server_key_iv: "09a3b7b4ec020f41bff5064013a0ce75"
server_key_version: "encryptionkey@0"
I20260929 18:41:52.191215 3185 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 18:41:52.193521 3200 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:41:52.194798 3185 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:41:52.194939 3185 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790707036041078-13712-0/minicluster-data/master-0/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790707036041078-13712-0/minicluster-data/master-0/wal
uuid: "43f00870ee9d49b98c538ccf6229e9c3"
format_stamp: "Formatted at 2026-09-29 18:41:52 on dist-test-slave-j1xs"
server_key: "20b61cfa3309a596ea8248119158022a"
server_key_iv: "09a3b7b4ec020f41bff5064013a0ce75"
server_key_version: "encryptionkey@0"
I20260929 18:41:52.195066 3185 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790707036041078-13712-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790707036041078-13712-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790707036041078-13712-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:41:52.204576 3185 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:41:52.207759 3185 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:41:52.208007 3185 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:41:52.214931 3185 rpc_server.cc:307] RPC server started. Bound to: 127.13.100.62:37687
I20260929 18:41:52.214956 3252 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.100.62:37687 every 8 connection(s)
I20260929 18:41:52.215799 3185 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
I20260929 18:41:52.218258 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 3185
I20260929 18:41:52.218384 13712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790707036041078-13712-0/minicluster-data/master-0/wal/instance
I20260929 18:41:52.218497 3253 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:41:52.218700 13712 external_mini_cluster.cc:1428] Setting key 0a9c36d019238fbcc0a8623bbb722800
I20260929 18:41:52.225529 3253 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43f00870ee9d49b98c538ccf6229e9c3: Bootstrap starting.
Sep 29 18:41:52 dist-test-slave-j1xs krb5kdc[3172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707312, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.100.62@KRBTEST.COM
I20260929 18:41:52.227865 3253 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 43f00870ee9d49b98c538ccf6229e9c3: Neither blocks nor log segments found. Creating new log.
I20260929 18:41:52.228724 3253 log.cc:824] T 00000000000000000000000000000000 P 43f00870ee9d49b98c538ccf6229e9c3: Log is configured to *not* fsync() on all Append() calls
I20260929 18:41:52.230567 3253 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43f00870ee9d49b98c538ccf6229e9c3: No bootstrap required, opened a new log
I20260929 18:41:52.232841 3256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:52.219981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56423 (local address 127.13.100.62:37687)
0929 18:41:52.220392 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0929 18:41:52.220412 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 18:41:52.220452 (+ 40us) server_negotiation.cc:407] Connection header received
0929 18:41:52.221205 (+ 753us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:41:52.221228 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:41:52.221648 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:41:52.222024 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:41:52.222719 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:52.224212 (+ 1493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:52.225046 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:52.225712 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:52.225867 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:52.228332 (+ 2465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:41:52.228338 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:41:52.228352 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:41:52.228381 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:41:52.230220 (+ 1839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:41:52.230660 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:41:52.230670 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:41:52.230674 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:41:52.230775 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:41:52.231051 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:41:52.231055 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:41:52.231057 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:41:52.231497 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:41:52.231701 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0929 18:41:52.231924 (+ 223us) server_negotiation.cc:299] Negotiation successful
0929 18:41:52.232200 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":118,"threads_started":1}
I20260929 18:41:52.234300 3253 raft_consensus.cc:348] T 00000000000000000000000000000000 P 43f00870ee9d49b98c538ccf6229e9c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43f00870ee9d49b98c538ccf6229e9c3" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 37687 } }
I20260929 18:41:52.234570 3253 raft_consensus.cc:374] T 00000000000000000000000000000000 P 43f00870ee9d49b98c538ccf6229e9c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:41:52.234661 3253 raft_consensus.cc:729] T 00000000000000000000000000000000 P 43f00870ee9d49b98c538ccf6229e9c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43f00870ee9d49b98c538ccf6229e9c3, State: Initialized, Role: FOLLOWER
I20260929 18:41:52.235150 3253 consensus_queue.cc:260] T 00000000000000000000000000000000 P 43f00870ee9d49b98c538ccf6229e9c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43f00870ee9d49b98c538ccf6229e9c3" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 37687 } }
I20260929 18:41:52.235319 3253 raft_consensus.cc:388] T 00000000000000000000000000000000 P 43f00870ee9d49b98c538ccf6229e9c3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:41:52.235385 3253 raft_consensus.cc:482] T 00000000000000000000000000000000 P 43f00870ee9d49b98c538ccf6229e9c3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:41:52.235483 3253 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 43f00870ee9d49b98c538ccf6229e9c3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:41:52.235618 13712 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:41:52.236352 3253 raft_consensus.cc:504] T 00000000000000000000000000000000 P 43f00870ee9d49b98c538ccf6229e9c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43f00870ee9d49b98c538ccf6229e9c3" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 37687 } }
I20260929 18:41:52.236676 3253 leader_election.cc:302] T 00000000000000000000000000000000 P 43f00870ee9d49b98c538ccf6229e9c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 43f00870ee9d49b98c538ccf6229e9c3; no voters:
I20260929 18:41:52.237022 3253 leader_election.cc:288] T 00000000000000000000000000000000 P 43f00870ee9d49b98c538ccf6229e9c3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:41:52.237131 3262 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 43f00870ee9d49b98c538ccf6229e9c3 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:41:52.237366 3262 raft_consensus.cc:686] T 00000000000000000000000000000000 P 43f00870ee9d49b98c538ccf6229e9c3 [term 1 LEADER]: Becoming Leader. State: Replica: 43f00870ee9d49b98c538ccf6229e9c3, State: Running, Role: LEADER
I20260929 18:41:52.237650 3262 consensus_queue.cc:237] T 00000000000000000000000000000000 P 43f00870ee9d49b98c538ccf6229e9c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43f00870ee9d49b98c538ccf6229e9c3" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 37687 } }
I20260929 18:41:52.238313 3253 sys_catalog.cc:564] T 00000000000000000000000000000000 P 43f00870ee9d49b98c538ccf6229e9c3 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:41:52.239521 3263 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43f00870ee9d49b98c538ccf6229e9c3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "43f00870ee9d49b98c538ccf6229e9c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43f00870ee9d49b98c538ccf6229e9c3" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 37687 } } }
I20260929 18:41:52.239645 3263 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43f00870ee9d49b98c538ccf6229e9c3 [sys.catalog]: This master's current role is: LEADER
I20260929 18:41:52.239997 3264 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43f00870ee9d49b98c538ccf6229e9c3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 43f00870ee9d49b98c538ccf6229e9c3. Latest consensus state: current_term: 1 leader_uuid: "43f00870ee9d49b98c538ccf6229e9c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43f00870ee9d49b98c538ccf6229e9c3" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 37687 } } }
I20260929 18:41:52.240091 3264 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43f00870ee9d49b98c538ccf6229e9c3 [sys.catalog]: This master's current role is: LEADER
W20260929 18:41:52.243582 3278 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 43f00870ee9d49b98c538ccf6229e9c3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:41:52.243662 3278 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:41:52.243736 3273 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:41:52.244566 3273 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:41:52.247155 3256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:52.237933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54253 (local address 127.13.100.62:37687)
0929 18:41:52.238028 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0929 18:41:52.238031 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:41:52.238047 (+ 16us) server_negotiation.cc:407] Connection header received
0929 18:41:52.238240 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:41:52.238244 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:41:52.238286 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:41:52.238371 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:41:52.240319 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:52.241116 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:52.241927 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:52.242258 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:52.242357 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:52.243941 (+ 1584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:41:52.243945 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:41:52.243947 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:41:52.243970 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:41:52.245411 (+ 1441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:41:52.245826 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:41:52.245831 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:41:52.245832 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:41:52.245986 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:41:52.246278 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:41:52.246281 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:41:52.246283 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:41:52.246487 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:41:52.246614 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0929 18:41:52.246904 (+ 290us) server_negotiation.cc:299] Negotiation successful
0929 18:41:52.247017 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260929 18:41:52.250120 3273 catalog_manager.cc:1347] Generated new cluster ID: 8a342f50671b4884a16608fead8fa086
I20260929 18:41:52.250191 3273 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:41:52.262416 3273 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:41:52.263319 3273 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:41:52.272738 3273 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 43f00870ee9d49b98c538ccf6229e9c3: Generated new TSK 0
I20260929 18:41:52.273361 3273 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:41:52.280992 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 3185
2026-09-29T18:41:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1393 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790707036041078-13712-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:41:52 dist-test-slave-j1xs krb5kdc[3284](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:41:52 dist-test-slave-j1xs krb5kdc[3284](info): set up 2 sockets
Sep 29 18:41:52 dist-test-slave-j1xs krb5kdc[3284](info): commencing operation
krb5kdc: starting...
W20260929 18:41:53.306078 13712 mini_kdc.cc:121] Time spent starting KDC: real 1.005s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790707036041078-13712-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790707036041078-13712-0/krb5kdc/test-user.keytab.
Sep 29 18:41:53 dist-test-slave-j1xs krb5kdc[3284](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707313, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:41:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:41:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.100.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.100.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.100.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.62.keytab.
Entry for principal kudu/127.13.100.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.62.keytab.
I20260929 18:41:53.424825 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790707036041078-13712-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790707036041078-13712-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790707036041078-13712-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.100.62:35499
--webserver_interface=127.13.100.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.62.keytab
--principal=kudu/127.13.100.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.100.20:45949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.100.62:35499
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790707036041078-13712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790707036041078-13712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790707036041078-13712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:41:53.532361 3297 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:41:53.532660 3297 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:41:53.532734 3297 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:41:53.536799 3297 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:41:53.536887 3297 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:41:53.536926 3297 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:41:53.536983 3297 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:41:53.537015 3297 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:41:53.542192 3297 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.100.20:45949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790707036041078-13712-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790707036041078-13712-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.100.62:35499
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.62.keytab
--principal=kudu/127.13.100.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.100.62:35499
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.100.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3297
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790707036041078-13712-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:41:53.543262 3297 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:41:53.544298 3297 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:41:53.552542 3305 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:41:53.552528 3303 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:41:53.552526 3302 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:41:53.552826 3297 server_base.cc:1034] running on GCE node
I20260929 18:41:53.553390 3297 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:41:53.554571 3297 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:41:53.555778 3297 hybrid_clock.cc:648] HybridClock initialized: now 1790707313555750 us; error 41 us; skew 500 ppm
Sep 29 18:41:53 dist-test-slave-j1xs krb5kdc[3284](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707313, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:41:53.558570 3297 init.cc:377] Logged in from keytab as kudu/127.13.100.62@KRBTEST.COM (short username kudu)
I20260929 18:41:53.559468 3297 webserver.cc:460] Webserver started at http://127.13.100.62:44023/ using document root <none> and password file <none>
I20260929 18:41:53.560024 3297 fs_manager.cc:359] Metadata directory not provided
I20260929 18:41:53.560118 3297 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:41:53.560348 3297 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:41:53.562402 3297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790707036041078-13712-0/minicluster-data/master-0/data/instance:
uuid: "b3b5da932c6f4d438b462fa33cc394d6"
format_stamp: "Formatted at 2026-09-29 18:41:53 on dist-test-slave-j1xs"
server_key: "9ab7dc4c4bb4015de8b42a77fc063b9f"
server_key_iv: "5cb72d80a7b07a94bd41c22e96b227b2"
server_key_version: "encryptionkey@0"
I20260929 18:41:53.563024 3297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790707036041078-13712-0/minicluster-data/master-0/wal/instance:
uuid: "b3b5da932c6f4d438b462fa33cc394d6"
format_stamp: "Formatted at 2026-09-29 18:41:53 on dist-test-slave-j1xs"
server_key: "9ab7dc4c4bb4015de8b42a77fc063b9f"
server_key_iv: "5cb72d80a7b07a94bd41c22e96b227b2"
server_key_version: "encryptionkey@0"
I20260929 18:41:53.566552 3297 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260929 18:41:53.568902 3312 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:41:53.570191 3297 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260929 18:41:53.570323 3297 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790707036041078-13712-0/minicluster-data/master-0/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790707036041078-13712-0/minicluster-data/master-0/wal
uuid: "b3b5da932c6f4d438b462fa33cc394d6"
format_stamp: "Formatted at 2026-09-29 18:41:53 on dist-test-slave-j1xs"
server_key: "9ab7dc4c4bb4015de8b42a77fc063b9f"
server_key_iv: "5cb72d80a7b07a94bd41c22e96b227b2"
server_key_version: "encryptionkey@0"
I20260929 18:41:53.570428 3297 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790707036041078-13712-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790707036041078-13712-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790707036041078-13712-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:41:53.586037 3297 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:41:53.588894 3297 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:41:53.589105 3297 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:41:53.595995 3297 rpc_server.cc:307] RPC server started. Bound to: 127.13.100.62:35499
I20260929 18:41:53.596014 3364 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.100.62:35499 every 8 connection(s)
I20260929 18:41:53.596907 3297 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
I20260929 18:41:53.599668 3365 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:41:53.600905 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 3297
I20260929 18:41:53.601058 13712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790707036041078-13712-0/minicluster-data/master-0/wal/instance
I20260929 18:41:53.601401 13712 external_mini_cluster.cc:1428] Setting key b09df666619e2b77c29e005dd62c11b5
I20260929 18:41:53.605993 3365 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3b5da932c6f4d438b462fa33cc394d6: Bootstrap starting.
I20260929 18:41:53.608242 3365 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b3b5da932c6f4d438b462fa33cc394d6: Neither blocks nor log segments found. Creating new log.
I20260929 18:41:53.608997 3365 log.cc:824] T 00000000000000000000000000000000 P b3b5da932c6f4d438b462fa33cc394d6: Log is configured to *not* fsync() on all Append() calls
I20260929 18:41:53.610358 3368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:53.602901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53245 (local address 127.13.100.62:35499)
0929 18:41:53.603349 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0929 18:41:53.603364 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 18:41:53.603413 (+ 49us) server_negotiation.cc:407] Connection header received
0929 18:41:53.604034 (+ 621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:41:53.604049 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:41:53.604366 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:41:53.604708 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:41:53.605152 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:53.606234 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:53.607110 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:53.607580 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:53.607744 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:53.608683 (+ 939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:41:53.608687 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:41:53.608699 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:41:53.609098 (+ 399us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:41:53.609167 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:41:53.609198 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:41:53.609303 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 18:41:53.609427 (+ 124us) server_negotiation.cc:299] Negotiation successful
0929 18:41:53.609633 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":139,"threads_started":1}
I20260929 18:41:53.611054 3365 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3b5da932c6f4d438b462fa33cc394d6: No bootstrap required, opened a new log
I20260929 18:41:53.613447 13712 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:41:53.615607 3365 raft_consensus.cc:348] T 00000000000000000000000000000000 P b3b5da932c6f4d438b462fa33cc394d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3b5da932c6f4d438b462fa33cc394d6" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 35499 } }
I20260929 18:41:53.615875 3365 raft_consensus.cc:374] T 00000000000000000000000000000000 P b3b5da932c6f4d438b462fa33cc394d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:41:53.615981 3365 raft_consensus.cc:729] T 00000000000000000000000000000000 P b3b5da932c6f4d438b462fa33cc394d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3b5da932c6f4d438b462fa33cc394d6, State: Initialized, Role: FOLLOWER
I20260929 18:41:53.616477 3365 consensus_queue.cc:260] T 00000000000000000000000000000000 P b3b5da932c6f4d438b462fa33cc394d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3b5da932c6f4d438b462fa33cc394d6" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 35499 } }
I20260929 18:41:53.616608 3365 raft_consensus.cc:388] T 00000000000000000000000000000000 P b3b5da932c6f4d438b462fa33cc394d6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:41:53.616701 3365 raft_consensus.cc:482] T 00000000000000000000000000000000 P b3b5da932c6f4d438b462fa33cc394d6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:41:53.616823 3365 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b3b5da932c6f4d438b462fa33cc394d6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:41:53.617787 3365 raft_consensus.cc:504] T 00000000000000000000000000000000 P b3b5da932c6f4d438b462fa33cc394d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3b5da932c6f4d438b462fa33cc394d6" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 35499 } }
I20260929 18:41:53.618167 3365 leader_election.cc:302] T 00000000000000000000000000000000 P b3b5da932c6f4d438b462fa33cc394d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b3b5da932c6f4d438b462fa33cc394d6; no voters:
I20260929 18:41:53.618484 3365 leader_election.cc:288] T 00000000000000000000000000000000 P b3b5da932c6f4d438b462fa33cc394d6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:41:53.618638 3375 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b3b5da932c6f4d438b462fa33cc394d6 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:41:53.618881 3375 raft_consensus.cc:686] T 00000000000000000000000000000000 P b3b5da932c6f4d438b462fa33cc394d6 [term 1 LEADER]: Becoming Leader. State: Replica: b3b5da932c6f4d438b462fa33cc394d6, State: Running, Role: LEADER
I20260929 18:41:53.619212 3375 consensus_queue.cc:237] T 00000000000000000000000000000000 P b3b5da932c6f4d438b462fa33cc394d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3b5da932c6f4d438b462fa33cc394d6" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 35499 } }
I20260929 18:41:53.619904 3365 sys_catalog.cc:564] T 00000000000000000000000000000000 P b3b5da932c6f4d438b462fa33cc394d6 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:41:53.620818 3368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:53.615934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48907 (local address 127.13.100.62:35499)
0929 18:41:53.616066 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 18:41:53.616072 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:41:53.616364 (+ 292us) server_negotiation.cc:407] Connection header received
0929 18:41:53.616519 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:41:53.616522 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:41:53.616569 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:41:53.616654 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:41:53.617061 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:53.617587 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:53.618575 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:53.618941 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:53.619059 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:53.620085 (+ 1026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:41:53.620089 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:41:53.620092 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:41:53.620106 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:41:53.620135 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:41:53.620150 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:41:53.620222 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 18:41:53.620548 (+ 326us) server_negotiation.cc:299] Negotiation successful
0929 18:41:53.620624 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 18:41:53.620999 3376 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3b5da932c6f4d438b462fa33cc394d6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b3b5da932c6f4d438b462fa33cc394d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3b5da932c6f4d438b462fa33cc394d6" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 35499 } } }
I20260929 18:41:53.621115 3376 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3b5da932c6f4d438b462fa33cc394d6 [sys.catalog]: This master's current role is: LEADER
I20260929 18:41:53.621399 3377 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3b5da932c6f4d438b462fa33cc394d6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b3b5da932c6f4d438b462fa33cc394d6. Latest consensus state: current_term: 1 leader_uuid: "b3b5da932c6f4d438b462fa33cc394d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3b5da932c6f4d438b462fa33cc394d6" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 35499 } } }
I20260929 18:41:53.621482 3377 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3b5da932c6f4d438b462fa33cc394d6 [sys.catalog]: This master's current role is: LEADER
W20260929 18:41:53.624672 3390 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b3b5da932c6f4d438b462fa33cc394d6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:41:53.624744 3390 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:41:53.624837 3384 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:41:53.625524 3384 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:41:53.630851 3384 catalog_manager.cc:1347] Generated new cluster ID: 748de09556a941dbbd788fc6cc04f7da
I20260929 18:41:53.630913 3384 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:41:53.642616 3384 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:41:53.643751 3384 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:41:53.649533 3384 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b3b5da932c6f4d438b462fa33cc394d6: Generated new TSK 0
I20260929 18:41:53.650266 3384 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:41:53.656384 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 3297
2026-09-29T18:41:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1374 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790707036041078-13712-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:41:53 dist-test-slave-j1xs krb5kdc[3396](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:41:53 dist-test-slave-j1xs krb5kdc[3396](info): set up 2 sockets
Sep 29 18:41:53 dist-test-slave-j1xs krb5kdc[3396](info): commencing operation
krb5kdc: starting...
W20260929 18:41:54.686175 13712 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.006s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790707036041078-13712-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790707036041078-13712-0/krb5kdc/test-user.keytab.
Sep 29 18:41:54 dist-test-slave-j1xs krb5kdc[3396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707314, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:41:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:41:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.100.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.100.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.100.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.62.keytab.
Entry for principal kudu/127.13.100.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.62.keytab.
I20260929 18:41:54.807300 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790707036041078-13712-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790707036041078-13712-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790707036041078-13712-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.100.62:46639
--webserver_interface=127.13.100.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.62.keytab
--principal=kudu/127.13.100.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.100.20:45591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.100.62:46639
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790707036041078-13712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790707036041078-13712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790707036041078-13712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:41:54.913184 3409 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:41:54.913470 3409 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:41:54.913546 3409 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:41:54.917457 3409 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:41:54.917546 3409 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:41:54.917583 3409 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:41:54.917639 3409 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:41:54.917678 3409 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:41:54.922752 3409 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.100.20:45591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790707036041078-13712-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790707036041078-13712-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.100.62:46639
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.62.keytab
--principal=kudu/127.13.100.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.100.62:46639
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.100.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3409
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790707036041078-13712-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:41:54.923755 3409 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:41:54.924711 3409 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:41:54.932904 3415 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:41:54.933049 3414 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:41:54.933013 3417 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:41:54.933024 3409 server_base.cc:1034] running on GCE node
I20260929 18:41:54.933650 3409 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:41:54.934806 3409 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:41:54.935992 3409 hybrid_clock.cc:648] HybridClock initialized: now 1790707314935974 us; error 35 us; skew 500 ppm
Sep 29 18:41:54 dist-test-slave-j1xs krb5kdc[3396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707314, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:41:54.938783 3409 init.cc:377] Logged in from keytab as kudu/127.13.100.62@KRBTEST.COM (short username kudu)
I20260929 18:41:54.939666 3409 webserver.cc:460] Webserver started at http://127.13.100.62:36017/ using document root <none> and password file <none>
I20260929 18:41:54.940199 3409 fs_manager.cc:359] Metadata directory not provided
I20260929 18:41:54.940260 3409 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:41:54.940460 3409 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:41:54.942509 3409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790707036041078-13712-0/minicluster-data/master-0/data/instance:
uuid: "c0403a90e2314b44a1eca804e573f823"
format_stamp: "Formatted at 2026-09-29 18:41:54 on dist-test-slave-j1xs"
server_key: "fdb579237dc2714f82a0399e5b500501"
server_key_iv: "f10e9968f2753a355f4d4cc9f15542f9"
server_key_version: "encryptionkey@0"
I20260929 18:41:54.943027 3409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790707036041078-13712-0/minicluster-data/master-0/wal/instance:
uuid: "c0403a90e2314b44a1eca804e573f823"
format_stamp: "Formatted at 2026-09-29 18:41:54 on dist-test-slave-j1xs"
server_key: "fdb579237dc2714f82a0399e5b500501"
server_key_iv: "f10e9968f2753a355f4d4cc9f15542f9"
server_key_version: "encryptionkey@0"
I20260929 18:41:54.946556 3409 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260929 18:41:54.948951 3424 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:41:54.950348 3409 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:41:54.950562 3409 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790707036041078-13712-0/minicluster-data/master-0/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790707036041078-13712-0/minicluster-data/master-0/wal
uuid: "c0403a90e2314b44a1eca804e573f823"
format_stamp: "Formatted at 2026-09-29 18:41:54 on dist-test-slave-j1xs"
server_key: "fdb579237dc2714f82a0399e5b500501"
server_key_iv: "f10e9968f2753a355f4d4cc9f15542f9"
server_key_version: "encryptionkey@0"
I20260929 18:41:54.950717 3409 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790707036041078-13712-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790707036041078-13712-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790707036041078-13712-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:41:54.955304 3409 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:41:54.958482 3409 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:41:54.958668 3409 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:41:54.965802 3409 rpc_server.cc:307] RPC server started. Bound to: 127.13.100.62:46639
I20260929 18:41:54.965826 3476 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.100.62:46639 every 8 connection(s)
I20260929 18:41:54.966729 3409 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
I20260929 18:41:54.969373 3477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:41:54.973067 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 3409
I20260929 18:41:54.973208 13712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790707036041078-13712-0/minicluster-data/master-0/wal/instance
I20260929 18:41:54.973501 13712 external_mini_cluster.cc:1428] Setting key d79f530957e85b65a88a13b4717a2f2b
I20260929 18:41:54.975162 3477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0403a90e2314b44a1eca804e573f823: Bootstrap starting.
I20260929 18:41:54.977404 3477 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c0403a90e2314b44a1eca804e573f823: Neither blocks nor log segments found. Creating new log.
I20260929 18:41:54.978219 3477 log.cc:824] T 00000000000000000000000000000000 P c0403a90e2314b44a1eca804e573f823: Log is configured to *not* fsync() on all Append() calls
I20260929 18:41:54.978884 3480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:54.974958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38093 (local address 127.13.100.62:46639)
0929 18:41:54.975442 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0929 18:41:54.975457 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 18:41:54.975493 (+ 36us) server_negotiation.cc:407] Connection header received
0929 18:41:54.976129 (+ 636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:41:54.976144 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:41:54.976416 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:41:54.976756 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:41:54.977514 (+ 758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:41:54.977519 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:41:54.977534 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:41:54.977548 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:41:54.977608 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:41:54.977628 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:41:54.977755 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0929 18:41:54.977852 (+ 97us) server_negotiation.cc:299] Negotiation successful
0929 18:41:54.978046 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":130,"threads_started":1}
I20260929 18:41:54.980074 3477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0403a90e2314b44a1eca804e573f823: No bootstrap required, opened a new log
I20260929 18:41:54.981541 13712 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:41:54.984087 3477 raft_consensus.cc:348] T 00000000000000000000000000000000 P c0403a90e2314b44a1eca804e573f823 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0403a90e2314b44a1eca804e573f823" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 46639 } }
I20260929 18:41:54.984402 3477 raft_consensus.cc:374] T 00000000000000000000000000000000 P c0403a90e2314b44a1eca804e573f823 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:41:54.984472 3477 raft_consensus.cc:729] T 00000000000000000000000000000000 P c0403a90e2314b44a1eca804e573f823 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0403a90e2314b44a1eca804e573f823, State: Initialized, Role: FOLLOWER
I20260929 18:41:54.984979 3477 consensus_queue.cc:260] T 00000000000000000000000000000000 P c0403a90e2314b44a1eca804e573f823 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0403a90e2314b44a1eca804e573f823" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 46639 } }
I20260929 18:41:54.985147 3477 raft_consensus.cc:388] T 00000000000000000000000000000000 P c0403a90e2314b44a1eca804e573f823 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:41:54.985179 3480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:54.983517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39951 (local address 127.13.100.62:46639)
0929 18:41:54.983649 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 18:41:54.983653 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:41:54.983832 (+ 179us) server_negotiation.cc:407] Connection header received
0929 18:41:54.983947 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:41:54.983949 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:41:54.983976 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:41:54.984038 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:41:54.984797 (+ 759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:41:54.984800 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:41:54.984802 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:41:54.984811 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:41:54.984835 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:41:54.984844 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:41:54.984916 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 18:41:54.985026 (+ 110us) server_negotiation.cc:299] Negotiation successful
0929 18:41:54.985055 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 18:41:54.985221 3477 raft_consensus.cc:482] T 00000000000000000000000000000000 P c0403a90e2314b44a1eca804e573f823 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:41:54.985327 3477 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c0403a90e2314b44a1eca804e573f823 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:41:54.986387 3477 raft_consensus.cc:504] T 00000000000000000000000000000000 P c0403a90e2314b44a1eca804e573f823 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0403a90e2314b44a1eca804e573f823" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 46639 } }
I20260929 18:41:54.986718 3477 leader_election.cc:302] T 00000000000000000000000000000000 P c0403a90e2314b44a1eca804e573f823 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c0403a90e2314b44a1eca804e573f823; no voters:
I20260929 18:41:54.987035 3477 leader_election.cc:288] T 00000000000000000000000000000000 P c0403a90e2314b44a1eca804e573f823 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:41:54.987159 3487 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c0403a90e2314b44a1eca804e573f823 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:41:54.987378 3487 raft_consensus.cc:686] T 00000000000000000000000000000000 P c0403a90e2314b44a1eca804e573f823 [term 1 LEADER]: Becoming Leader. State: Replica: c0403a90e2314b44a1eca804e573f823, State: Running, Role: LEADER
I20260929 18:41:54.987713 3487 consensus_queue.cc:237] T 00000000000000000000000000000000 P c0403a90e2314b44a1eca804e573f823 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0403a90e2314b44a1eca804e573f823" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 46639 } }
I20260929 18:41:54.988206 3477 sys_catalog.cc:564] T 00000000000000000000000000000000 P c0403a90e2314b44a1eca804e573f823 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:41:54.989442 3489 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0403a90e2314b44a1eca804e573f823 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c0403a90e2314b44a1eca804e573f823. Latest consensus state: current_term: 1 leader_uuid: "c0403a90e2314b44a1eca804e573f823" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0403a90e2314b44a1eca804e573f823" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 46639 } } }
I20260929 18:41:54.989419 3488 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0403a90e2314b44a1eca804e573f823 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c0403a90e2314b44a1eca804e573f823" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0403a90e2314b44a1eca804e573f823" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 46639 } } }
I20260929 18:41:54.989563 3488 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0403a90e2314b44a1eca804e573f823 [sys.catalog]: This master's current role is: LEADER
I20260929 18:41:54.989563 3489 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0403a90e2314b44a1eca804e573f823 [sys.catalog]: This master's current role is: LEADER
I20260929 18:41:54.990015 3497 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:41:54.993005 3497 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:41:54.998154 3497 catalog_manager.cc:1347] Generated new cluster ID: a0894f5d7127443a9c3536e9b01bacfa
I20260929 18:41:54.998229 3497 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:41:55.017110 3497 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:41:55.017513 3497 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:41:55.035650 3497 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c0403a90e2314b44a1eca804e573f823: Generated new TSK 0
I20260929 18:41:55.036353 3497 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:41:55.051582 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 3409
2026-09-29T18:41:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1395 ms)
[----------] 18 tests from AuthTokenIssuingTest (25281 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707036041078-13712-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:41:55 dist-test-slave-j1xs krb5kdc[3508](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:41:55 dist-test-slave-j1xs krb5kdc[3508](info): set up 2 sockets
Sep 29 18:41:55 dist-test-slave-j1xs krb5kdc[3508](info): commencing operation
krb5kdc: starting...
W20260929 18:41:56.094218 13712 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707036041078-13712-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707036041078-13712-0/krb5kdc/test-user.keytab.
Sep 29 18:41:56 dist-test-slave-j1xs krb5kdc[3508](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707316, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:41:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:41:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.100.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.100.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.100.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.62.keytab.
Entry for principal kudu/127.13.100.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.62.keytab.
I20260929 18:41:56.214942 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707036041078-13712-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707036041078-13712-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707036041078-13712-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.100.62:37443
--webserver_interface=127.13.100.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.62.keytab
--principal=kudu/127.13.100.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.100.20:32803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.100.62:37443,127.13.100.61:41479,127.13.100.60:34553
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707036041078-13712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707036041078-13712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707036041078-13712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:41:56.321115 3521 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 18:41:56.321457 3521 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:41:56.321511 3521 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:41:56.321583 3521 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:41:56.325672 3521 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:41:56.325757 3521 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:41:56.325795 3521 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:41:56.325832 3521 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:41:56.325863 3521 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:41:56.331075 3521 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.100.20:32803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707036041078-13712-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707036041078-13712-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.100.62:37443,127.13.100.61:41479,127.13.100.60:34553
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.62.keytab
--principal=kudu/127.13.100.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.100.62:37443
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.100.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3521
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707036041078-13712-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:41:56.332144 3521 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:41:56.333148 3521 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:41:56.341545 3529 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:41:56.341570 3526 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:41:56.341570 3527 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:41:56.342080 3521 server_base.cc:1034] running on GCE node
I20260929 18:41:56.342638 3521 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:41:56.343734 3521 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:41:56.344933 3521 hybrid_clock.cc:648] HybridClock initialized: now 1790707316344917 us; error 37 us; skew 500 ppm
Sep 29 18:41:56 dist-test-slave-j1xs krb5kdc[3508](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707316, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:41:56.347746 3521 init.cc:377] Logged in from keytab as kudu/127.13.100.62@KRBTEST.COM (short username kudu)
I20260929 18:41:56.348587 3521 webserver.cc:460] Webserver started at http://127.13.100.62:42141/ using document root <none> and password file <none>
I20260929 18:41:56.349172 3521 fs_manager.cc:359] Metadata directory not provided
I20260929 18:41:56.349269 3521 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:41:56.349503 3521 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:41:56.351588 3521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707036041078-13712-0/minicluster-data/master-0/data/instance:
uuid: "96303e1b5ab849f18213c81e071494b1"
format_stamp: "Formatted at 2026-09-29 18:41:56 on dist-test-slave-j1xs"
server_key: "d569817d9da84a38bd893d71379c5220"
server_key_iv: "de46e46bb2a51f83b977045d171ff163"
server_key_version: "encryptionkey@0"
I20260929 18:41:56.352162 3521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707036041078-13712-0/minicluster-data/master-0/wal/instance:
uuid: "96303e1b5ab849f18213c81e071494b1"
format_stamp: "Formatted at 2026-09-29 18:41:56 on dist-test-slave-j1xs"
server_key: "d569817d9da84a38bd893d71379c5220"
server_key_iv: "de46e46bb2a51f83b977045d171ff163"
server_key_version: "encryptionkey@0"
I20260929 18:41:56.355885 3521 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260929 18:41:56.358368 3536 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:41:56.359776 3521 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:41:56.359942 3521 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707036041078-13712-0/minicluster-data/master-0/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707036041078-13712-0/minicluster-data/master-0/wal
uuid: "96303e1b5ab849f18213c81e071494b1"
format_stamp: "Formatted at 2026-09-29 18:41:56 on dist-test-slave-j1xs"
server_key: "d569817d9da84a38bd893d71379c5220"
server_key_iv: "de46e46bb2a51f83b977045d171ff163"
server_key_version: "encryptionkey@0"
I20260929 18:41:56.360072 3521 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707036041078-13712-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707036041078-13712-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707036041078-13712-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:41:56.385039 3521 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:41:56.388461 3521 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:41:56.388629 3521 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:41:56.395824 3521 rpc_server.cc:307] RPC server started. Bound to: 127.13.100.62:37443
I20260929 18:41:56.396121 3588 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.100.62:37443 every 8 connection(s)
I20260929 18:41:56.396844 3521 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
I20260929 18:41:56.399540 3589 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:41:56.400991 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 3521
I20260929 18:41:56.401141 13712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707036041078-13712-0/minicluster-data/master-0/wal/instance
I20260929 18:41:56.401451 13712 external_mini_cluster.cc:1428] Setting key ff43ab57b782601297a3175b1db6780a
I20260929 18:41:56.403615 3589 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.100.62" port: 37443 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.13.100.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.100.61@KRBTEST.COM" created.
Sep 29 18:41:56 dist-test-slave-j1xs krb5kdc[3508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707316, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.62@KRBTEST.COM for kudu/127.13.100.62@KRBTEST.COM
I20260929 18:41:56.420302 3593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:56.406307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42200 (local address 127.13.100.62:37443)
0929 18:41:56.406851 (+ 544us) server_negotiation.cc:206] Beginning negotiation
0929 18:41:56.406873 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0929 18:41:56.407083 (+ 210us) server_negotiation.cc:407] Connection header received
0929 18:41:56.408468 (+ 1385us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:41:56.408472 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:41:56.408647 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:41:56.408798 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:41:56.410137 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:56.411217 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:56.412287 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:56.412775 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:56.412994 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:56.415290 (+ 2296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:41:56.415295 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:41:56.415305 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:41:56.415323 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:41:56.417113 (+ 1790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:41:56.417716 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:41:56.417720 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:41:56.417722 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:41:56.417823 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:41:56.418210 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:41:56.418212 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:41:56.418213 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:41:56.418603 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:41:56.418706 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 18:41:56.419194 (+ 488us) server_negotiation.cc:299] Negotiation successful
0929 18:41:56.419365 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":48,"server-negotiator.queue_time_us":427,"spinlock_wait_cycles":50432,"thread_start_us":146,"threads_started":1}
I20260929 18:41:56.420308 3592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:56.406307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:37443 (local address 127.0.0.1:42200)
0929 18:41:56.406885 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:56.406941 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0929 18:41:56.408086 (+ 1145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:41:56.408857 (+ 771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:41:56.408866 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:41:56.409432 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:41:56.409798 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:41:56.409815 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:41:56.411348 (+ 1533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:41:56.411351 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:41:56.412167 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:41:56.412176 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:41:56.412970 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:41:56.412973 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:41:56.413097 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:56.413711 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:41:56.413726 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:41:56.415144 (+ 1418us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:41:56.417230 (+ 2086us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:41:56.417242 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:41:56.417254 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:41:56.417594 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:41:56.417945 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:41:56.417950 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:41:56.417953 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:41:56.418105 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:41:56.418724 (+ 619us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:41:56.418731 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:41:56.418895 (+ 164us) client_negotiation.cc:769] Sending connection context
0929 18:41:56.419078 (+ 183us) client_negotiation.cc:240] Negotiation successful
0929 18:41:56.419465 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"mutex_wait_us":79,"thread_start_us":145,"threads_started":1}
I20260929 18:41:56.423724 3589 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.100.61" port: 41479 } has no permanent_uuid. Determining permanent_uuid...
I20260929 18:41:56.424396 3592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:56.424166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.61:41479 (local address 127.0.0.1:41432)
0929 18:41:56.424276 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:56.424341 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.61:41479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260929 18:41:56.424590 3539 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.100.61:41479: connect: Connection refused (error 111)
I20260929 18:41:56.425333 3592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:56.425126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.61:41479 (local address 127.0.0.1:41434)
0929 18:41:56.425236 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:56.425287 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.61:41479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260929 18:41:56.425624 3589 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.100.61:41479: Network error: Client connection negotiation failed: client connection to 127.13.100.61:41479: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.100.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.61.keytab.
Entry for principal kudu/127.13.100.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.61.keytab.
I20260929 18:41:56.430017 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707036041078-13712-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707036041078-13712-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707036041078-13712-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707036041078-13712-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.100.61:41479
--webserver_interface=127.13.100.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.61.keytab
--principal=kudu/127.13.100.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.100.20:32803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.100.62:37443,127.13.100.61:41479,127.13.100.60:34553
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707036041078-13712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707036041078-13712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707036041078-13712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 18:41:56.474906 3589 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.100.61" port: 41479 } attempt: 1
I20260929 18:41:56.475736 3592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:56.475440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.61:41479 (local address 127.0.0.1:41438)
0929 18:41:56.475597 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:56.475659 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.61:41479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260929 18:41:56.476472 3592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:56.476235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.61:41479 (local address 127.0.0.1:41450)
0929 18:41:56.476363 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:56.476419 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.61:41479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260929 18:41:56.476663 3589 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.100.61:41479: Network error: Client connection negotiation failed: client connection to 127.13.100.61:41479: connect: Connection refused (error 111)
W20260929 18:41:56.536156 3596 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 18:41:56.536417 3596 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:41:56.536439 3596 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:41:56.536532 3596 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:41:56.540788 3596 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:41:56.540849 3596 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:41:56.540870 3596 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:41:56.540887 3596 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:41:56.540935 3596 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:41:56.544854 3589 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.100.61" port: 41479 } attempt: 2
I20260929 18:41:56.545725 3592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:56.545438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.61:41479 (local address 127.0.0.1:41456)
0929 18:41:56.545594 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:56.545654 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.61:41479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260929 18:41:56.546435 3592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:56.546229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.61:41479 (local address 127.0.0.1:41462)
0929 18:41:56.546332 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:56.546382 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.61:41479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260929 18:41:56.546567 3589 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.100.61:41479: Network error: Client connection negotiation failed: client connection to 127.13.100.61:41479: connect: Connection refused (error 111)
I20260929 18:41:56.546583 3596 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.100.20:32803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707036041078-13712-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707036041078-13712-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.13.100.62:37443,127.13.100.61:41479,127.13.100.60:34553
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.61.keytab
--principal=kudu/127.13.100.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.100.61:41479
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707036041078-13712-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.13.100.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3596
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707036041078-13712-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:41:56.547628 3596 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:41:56.548601 3596 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:41:56.556792 3604 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:41:56.556818 3602 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:41:56.556792 3601 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:41:56.557206 3596 server_base.cc:1034] running on GCE node
I20260929 18:41:56.557772 3596 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:41:56.558845 3596 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:41:56.560009 3596 hybrid_clock.cc:648] HybridClock initialized: now 1790707316559979 us; error 42 us; skew 500 ppm
Sep 29 18:41:56 dist-test-slave-j1xs krb5kdc[3508](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707316, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:41:56.562870 3596 init.cc:377] Logged in from keytab as kudu/127.13.100.61@KRBTEST.COM (short username kudu)
I20260929 18:41:56.563786 3596 webserver.cc:460] Webserver started at http://127.13.100.61:42265/ using document root <none> and password file <none>
I20260929 18:41:56.564374 3596 fs_manager.cc:359] Metadata directory not provided
I20260929 18:41:56.564461 3596 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:41:56.564662 3596 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:41:56.566748 3596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707036041078-13712-0/minicluster-data/master-1/data/instance:
uuid: "1a4b65aa97e14df392a22dff75dca5ac"
format_stamp: "Formatted at 2026-09-29 18:41:56 on dist-test-slave-j1xs"
server_key: "047e1f1cbdbf7e38a5203db046459556"
server_key_iv: "2cc3be18b9ce3e88d6d6c0aed11cc27e"
server_key_version: "encryptionkey@0"
I20260929 18:41:56.567298 3596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707036041078-13712-0/minicluster-data/master-1/wal/instance:
uuid: "1a4b65aa97e14df392a22dff75dca5ac"
format_stamp: "Formatted at 2026-09-29 18:41:56 on dist-test-slave-j1xs"
server_key: "047e1f1cbdbf7e38a5203db046459556"
server_key_iv: "2cc3be18b9ce3e88d6d6c0aed11cc27e"
server_key_version: "encryptionkey@0"
I20260929 18:41:56.570948 3596 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260929 18:41:56.573319 3611 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:41:56.574584 3596 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260929 18:41:56.574723 3596 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707036041078-13712-0/minicluster-data/master-1/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707036041078-13712-0/minicluster-data/master-1/wal
uuid: "1a4b65aa97e14df392a22dff75dca5ac"
format_stamp: "Formatted at 2026-09-29 18:41:56 on dist-test-slave-j1xs"
server_key: "047e1f1cbdbf7e38a5203db046459556"
server_key_iv: "2cc3be18b9ce3e88d6d6c0aed11cc27e"
server_key_version: "encryptionkey@0"
I20260929 18:41:56.574832 3596 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707036041078-13712-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707036041078-13712-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707036041078-13712-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:41:56.599938 3596 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:41:56.602890 3596 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:41:56.603120 3596 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:41:56.610137 3596 rpc_server.cc:307] RPC server started. Bound to: 127.13.100.61:41479
I20260929 18:41:56.610458 3663 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.100.61:41479 every 8 connection(s)
I20260929 18:41:56.611088 3596 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707036041078-13712-0/minicluster-data/master-1/data/info.pb
I20260929 18:41:56.613699 3664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:41:56.616247 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 3596
I20260929 18:41:56.616391 13712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707036041078-13712-0/minicluster-data/master-1/wal/instance
I20260929 18:41:56.616660 13712 external_mini_cluster.cc:1428] Setting key 2e543536979554128f0a179a6c6fbf7c
I20260929 18:41:56.617377 3664 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.100.62" port: 37443 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.13.100.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.100.60@KRBTEST.COM" created.
Sep 29 18:41:56 dist-test-slave-j1xs krb5kdc[3508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707316, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.61@KRBTEST.COM for kudu/127.13.100.62@KRBTEST.COM
I20260929 18:41:56.631397 3593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:56.619344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42202 (local address 127.13.100.62:37443)
0929 18:41:56.619481 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0929 18:41:56.619484 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:41:56.620413 (+ 929us) server_negotiation.cc:407] Connection header received
0929 18:41:56.621363 (+ 950us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:41:56.621367 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:41:56.621428 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:41:56.621536 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:41:56.622687 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:56.623214 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:56.624379 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:56.624626 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:56.624741 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:56.627777 (+ 3036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:41:56.627780 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:41:56.627782 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:41:56.627805 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:41:56.629304 (+ 1499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:41:56.629809 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:41:56.629813 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:41:56.629815 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:41:56.629963 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:41:56.630301 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:41:56.630305 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:41:56.630307 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:41:56.630489 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:41:56.630596 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 18:41:56.631096 (+ 500us) server_negotiation.cc:299] Negotiation successful
0929 18:41:56.631218 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 18:41:56.632156 3667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:56.619765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:37443 (local address 127.0.0.1:42202)
0929 18:41:56.620268 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:56.620308 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 18:41:56.621171 (+ 863us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:41:56.621695 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:41:56.621702 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:41:56.622094 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:41:56.622550 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:41:56.622563 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:41:56.623341 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:41:56.623345 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:41:56.624261 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:41:56.624271 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:41:56.624778 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:41:56.624783 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:41:56.624948 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:56.625534 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:41:56.625551 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:41:56.627589 (+ 2038us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:41:56.629430 (+ 1841us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:41:56.629435 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:41:56.629449 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:41:56.629703 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:41:56.630061 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:41:56.630064 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:41:56.630066 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:41:56.630203 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:41:56.630595 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:41:56.630602 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:41:56.630849 (+ 247us) client_negotiation.cc:769] Sending connection context
0929 18:41:56.631032 (+ 183us) client_negotiation.cc:240] Negotiation successful
0929 18:41:56.631340 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":149,"threads_started":1}
I20260929 18:41:56.633738 3664 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.100.61" port: 41479 } has no permanent_uuid. Determining permanent_uuid...
I20260929 18:41:56.637732 3589 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.100.61" port: 41479 } attempt: 3
Sep 29 18:41:56 dist-test-slave-j1xs krb5kdc[3508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707316, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.61@KRBTEST.COM for kudu/127.13.100.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.100.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.60.keytab.
Entry for principal kudu/127.13.100.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.60.keytab.
Sep 29 18:41:56 dist-test-slave-j1xs krb5kdc[3508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707316, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.62@KRBTEST.COM for kudu/127.13.100.61@KRBTEST.COM
I20260929 18:41:56.643183 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707036041078-13712-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707036041078-13712-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707036041078-13712-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707036041078-13712-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.100.60:34553
--webserver_interface=127.13.100.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.60.keytab
--principal=kudu/127.13.100.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.100.20:32803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.100.62:37443,127.13.100.61:41479,127.13.100.60:34553
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707036041078-13712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707036041078-13712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707036041078-13712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 18:41:56.643507 3667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:56.634246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.61:41479 (local address 127.0.0.1:41468)
0929 18:41:56.634370 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:56.634381 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 18:41:56.634443 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:41:56.635003 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:41:56.635006 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:41:56.635139 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:41:56.635235 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:41:56.635240 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:41:56.636356 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:41:56.636362 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:41:56.636897 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:41:56.636905 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:41:56.637558 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:41:56.637561 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:41:56.637645 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:56.638203 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:41:56.638225 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:41:56.639574 (+ 1349us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:41:56.641731 (+ 2157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:41:56.641737 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:41:56.641742 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:41:56.642054 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:41:56.642395 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:41:56.642400 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:41:56.642404 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:41:56.642485 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:41:56.643078 (+ 593us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:41:56.643084 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:41:56.643178 (+ 94us) client_negotiation.cc:769] Sending connection context
0929 18:41:56.643281 (+ 103us) client_negotiation.cc:240] Negotiation successful
0929 18:41:56.643392 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260929 18:41:56.643977 3669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:56.634375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41468 (local address 127.13.100.61:41479)
0929 18:41:56.634662 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0929 18:41:56.634670 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 18:41:56.634689 (+ 19us) server_negotiation.cc:407] Connection header received
0929 18:41:56.634722 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:41:56.634725 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:41:56.634865 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:41:56.634966 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:41:56.635418 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:56.636246 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:56.637016 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:56.637453 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:56.637571 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:56.639707 (+ 2136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:41:56.639717 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:41:56.639731 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:41:56.639754 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:41:56.641605 (+ 1851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:41:56.642189 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:41:56.642196 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:41:56.642198 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:41:56.642276 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:41:56.642598 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:41:56.642602 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:41:56.642605 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:41:56.642946 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:41:56.643095 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0929 18:41:56.643740 (+ 645us) server_negotiation.cc:299] Negotiation successful
0929 18:41:56.643856 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":103,"threads_started":1}
I20260929 18:41:56.646032 3664 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.100.60" port: 34553 } has no permanent_uuid. Determining permanent_uuid...
I20260929 18:41:56.646689 3667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:56.646465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.60:34553 (local address 127.0.0.1:37012)
0929 18:41:56.646568 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:56.646643 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.60:34553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260929 18:41:56.646864 3614 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.100.60:34553: connect: Connection refused (error 111)
I20260929 18:41:56.647608 3667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:56.647414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.60:34553 (local address 127.0.0.1:37026)
0929 18:41:56.647513 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:56.647556 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.60:34553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260929 18:41:56.647753 3664 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.100.60:34553: Network error: Client connection negotiation failed: client connection to 127.13.100.60:34553: connect: Connection refused (error 111)
I20260929 18:41:56.659910 3592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:56.638218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.61:41479 (local address 127.0.0.1:41484)
0929 18:41:56.638338 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:56.638356 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 18:41:56.638474 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:41:56.638739 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:41:56.638742 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:41:56.638898 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:41:56.638974 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:41:56.638979 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:41:56.640085 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:41:56.640089 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:41:56.640620 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:41:56.640627 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:41:56.641089 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:41:56.641093 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:41:56.641198 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:56.641659 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:41:56.641670 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:41:56.643171 (+ 1501us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:41:56.658484 (+ 15313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:41:56.658488 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:41:56.658489 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:41:56.658720 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:41:56.659019 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:41:56.659021 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:41:56.659023 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:41:56.659099 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:41:56.659507 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:41:56.659508 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:41:56.659602 (+ 94us) client_negotiation.cc:769] Sending connection context
0929 18:41:56.659680 (+ 78us) client_negotiation.cc:240] Negotiation successful
0929 18:41:56.659791 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260929 18:41:56.660040 3670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:56.638273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41484 (local address 127.13.100.61:41479)
0929 18:41:56.638549 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0929 18:41:56.638559 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 18:41:56.638570 (+ 11us) server_negotiation.cc:407] Connection header received
0929 18:41:56.638603 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:41:56.638605 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:41:56.638635 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:41:56.638724 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:41:56.639081 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:56.639919 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:56.640735 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:56.640978 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:56.641089 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:56.643320 (+ 2231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:41:56.643326 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:41:56.643329 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:41:56.643356 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:41:56.658319 (+ 14963us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:41:56.658826 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:41:56.658832 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:41:56.658835 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:41:56.658929 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:41:56.659184 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:41:56.659187 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:41:56.659189 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:41:56.659420 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:41:56.659514 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0929 18:41:56.659751 (+ 237us) server_negotiation.cc:299] Negotiation successful
0929 18:41:56.659871 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":86,"threads_started":1}
I20260929 18:41:56.660773 3589 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.100.60" port: 34553 } has no permanent_uuid. Determining permanent_uuid...
I20260929 18:41:56.661434 3592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:56.661225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.60:34553 (local address 127.0.0.1:37042)
0929 18:41:56.661337 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:56.661381 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.60:34553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260929 18:41:56.662093 3592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:56.661847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.60:34553 (local address 127.0.0.1:37050)
0929 18:41:56.661997 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:56.662040 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.60:34553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
W20260929 18:41:56.662246 3589 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.100.60:34553: Network error: Client connection negotiation failed: client connection to 127.13.100.60:34553: connect: Connection refused (error 111)
I20260929 18:41:56.693414 3589 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.100.60" port: 34553 } attempt: 1
I20260929 18:41:56.694425 3592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:56.694095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.60:34553 (local address 127.0.0.1:37052)
0929 18:41:56.694287 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:56.694350 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.60:34553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260929 18:41:56.695194 3592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:56.694965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.60:34553 (local address 127.0.0.1:37064)
0929 18:41:56.695086 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:56.695139 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.60:34553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260929 18:41:56.695329 3589 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.100.60:34553: Network error: Client connection negotiation failed: client connection to 127.13.100.60:34553: connect: Connection refused (error 111)
I20260929 18:41:56.697001 3664 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.100.60" port: 34553 } attempt: 1
I20260929 18:41:56.697734 3667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:56.697484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.60:34553 (local address 127.0.0.1:37072)
0929 18:41:56.697620 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:56.697671 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.60:34553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260929 18:41:56.698427 3667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:56.698216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.60:34553 (local address 127.0.0.1:37074)
0929 18:41:56.698324 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:56.698375 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.60:34553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260929 18:41:56.698566 3664 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.100.60:34553: Network error: Client connection negotiation failed: client connection to 127.13.100.60:34553: connect: Connection refused (error 111)
W20260929 18:41:56.756944 3671 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 18:41:56.757253 3671 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:41:56.757326 3671 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:41:56.757409 3671 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:41:56.761485 3671 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:41:56.761571 3671 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:41:56.761610 3671 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:41:56.761667 3671 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:41:56.761708 3671 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:41:56.766727 3664 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.100.60" port: 34553 } attempt: 2
I20260929 18:41:56.767093 3671 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.100.20:32803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707036041078-13712-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707036041078-13712-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.13.100.62:37443,127.13.100.61:41479,127.13.100.60:34553
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.60.keytab
--principal=kudu/127.13.100.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.100.60:34553
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707036041078-13712-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.13.100.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3671
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707036041078-13712-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:41:56.767560 3667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:56.767269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.60:34553 (local address 127.0.0.1:37082)
0929 18:41:56.767436 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:56.767489 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.60:34553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260929 18:41:56.768150 3671 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:41:56.768306 3667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:56.768068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.60:34553 (local address 127.0.0.1:37098)
0929 18:41:56.768174 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:56.768226 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.60:34553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260929 18:41:56.768441 3664 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.100.60:34553: Network error: Client connection negotiation failed: client connection to 127.13.100.60:34553: connect: Connection refused (error 111)
I20260929 18:41:56.769258 3671 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 18:41:56.770514 3589 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.100.60" port: 34553 } attempt: 2
I20260929 18:41:56.771194 3592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:56.770885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.60:34553 (local address 127.0.0.1:37108)
0929 18:41:56.771054 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:56.771117 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.60:34553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260929 18:41:56.772330 3592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:56.772097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.60:34553 (local address 127.0.0.1:37124)
0929 18:41:56.772216 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:56.772268 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.60:34553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260929 18:41:56.772465 3589 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.100.60:34553: Network error: Client connection negotiation failed: client connection to 127.13.100.60:34553: connect: Connection refused (error 111)
W20260929 18:41:56.777856 3677 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:41:56.777873 3678 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:41:56.778103 3671 server_base.cc:1034] running on GCE node
W20260929 18:41:56.777918 3680 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:41:56.778717 3671 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:41:56.779785 3671 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:41:56.780947 3671 hybrid_clock.cc:648] HybridClock initialized: now 1790707316780927 us; error 35 us; skew 500 ppm
Sep 29 18:41:56 dist-test-slave-j1xs krb5kdc[3508](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707316, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:41:56.783612 3671 init.cc:377] Logged in from keytab as kudu/127.13.100.60@KRBTEST.COM (short username kudu)
I20260929 18:41:56.784505 3671 webserver.cc:460] Webserver started at http://127.13.100.60:42299/ using document root <none> and password file <none>
I20260929 18:41:56.785041 3671 fs_manager.cc:359] Metadata directory not provided
I20260929 18:41:56.785133 3671 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:41:56.785352 3671 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:41:56.787313 3671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707036041078-13712-0/minicluster-data/master-2/data/instance:
uuid: "852636c4803e4dbca073acbb031b783c"
format_stamp: "Formatted at 2026-09-29 18:41:56 on dist-test-slave-j1xs"
server_key: "5b6176b614be2d1653bc90d101fc43ac"
server_key_iv: "39f22c031696745aaf63a6035500bddc"
server_key_version: "encryptionkey@0"
I20260929 18:41:56.787864 3671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707036041078-13712-0/minicluster-data/master-2/wal/instance:
uuid: "852636c4803e4dbca073acbb031b783c"
format_stamp: "Formatted at 2026-09-29 18:41:56 on dist-test-slave-j1xs"
server_key: "5b6176b614be2d1653bc90d101fc43ac"
server_key_iv: "39f22c031696745aaf63a6035500bddc"
server_key_version: "encryptionkey@0"
I20260929 18:41:56.791745 3671 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 18:41:56.794291 3687 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:41:56.795549 3671 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:41:56.795758 3671 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707036041078-13712-0/minicluster-data/master-2/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707036041078-13712-0/minicluster-data/master-2/wal
uuid: "852636c4803e4dbca073acbb031b783c"
format_stamp: "Formatted at 2026-09-29 18:41:56 on dist-test-slave-j1xs"
server_key: "5b6176b614be2d1653bc90d101fc43ac"
server_key_iv: "39f22c031696745aaf63a6035500bddc"
server_key_version: "encryptionkey@0"
I20260929 18:41:56.795878 3671 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707036041078-13712-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707036041078-13712-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707036041078-13712-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:41:56.817615 3671 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:41:56.821025 3671 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:41:56.821249 3671 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:41:56.828382 3671 rpc_server.cc:307] RPC server started. Bound to: 127.13.100.60:34553
I20260929 18:41:56.828677 3739 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.100.60:34553 every 8 connection(s)
I20260929 18:41:56.829357 3671 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707036041078-13712-0/minicluster-data/master-2/data/info.pb
I20260929 18:41:56.830279 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 3671
I20260929 18:41:56.830416 13712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707036041078-13712-0/minicluster-data/master-2/wal/instance
I20260929 18:41:56.830658 13712 external_mini_cluster.cc:1428] Setting key 714b5c9c3e94073c7996bafb2bd66986
I20260929 18:41:56.832499 3740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:41:56.837047 3740 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.100.62" port: 37443 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 18:41:56 dist-test-slave-j1xs krb5kdc[3508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707316, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.100.60@KRBTEST.COM
Sep 29 18:41:56 dist-test-slave-j1xs krb5kdc[3508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707316, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.60@KRBTEST.COM for kudu/127.13.100.62@KRBTEST.COM
I20260929 18:41:56.846115 3743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:56.832030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37140 (local address 127.13.100.60:34553)
0929 18:41:56.832434 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0929 18:41:56.832453 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 18:41:56.832496 (+ 43us) server_negotiation.cc:407] Connection header received
0929 18:41:56.833321 (+ 825us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:41:56.833342 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:41:56.833731 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:41:56.834110 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:41:56.834792 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:56.836397 (+ 1605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:56.837378 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:56.838108 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:56.838268 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:56.841013 (+ 2745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:41:56.841024 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:41:56.841033 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:41:56.841059 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:41:56.842949 (+ 1890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:41:56.843483 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:41:56.843488 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:41:56.843492 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:41:56.843609 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:41:56.843916 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:41:56.843918 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:41:56.843919 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:41:56.844270 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:41:56.844493 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0929 18:41:56.844816 (+ 323us) server_negotiation.cc:299] Negotiation successful
0929 18:41:56.845091 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":110,"threads_started":1}
I20260929 18:41:56.848337 3744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:56.839122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:37443 (local address 127.0.0.1:42212)
0929 18:41:56.839703 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:56.839754 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 18:41:56.839897 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:41:56.840206 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:41:56.840210 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:41:56.840608 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:41:56.840740 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:41:56.840747 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:41:56.841584 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:41:56.841587 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:41:56.842265 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:41:56.842274 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:41:56.842713 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:41:56.842717 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:41:56.842890 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:56.843403 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:41:56.843421 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:41:56.844784 (+ 1363us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:41:56.846672 (+ 1888us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:41:56.846680 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:41:56.846685 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:41:56.846896 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:41:56.847202 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:41:56.847205 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:41:56.847207 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:41:56.847286 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:41:56.847653 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:41:56.847659 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:41:56.847788 (+ 129us) client_negotiation.cc:769] Sending connection context
0929 18:41:56.847962 (+ 174us) client_negotiation.cc:240] Negotiation successful
0929 18:41:56.848143 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":339,"threads_started":1}
I20260929 18:41:56.848371 3593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:56.839187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42212 (local address 127.13.100.62:37443)
0929 18:41:56.839420 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0929 18:41:56.839427 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 18:41:56.839859 (+ 432us) server_negotiation.cc:407] Connection header received
0929 18:41:56.840011 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:41:56.840015 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:41:56.840066 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:41:56.840153 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:41:56.840992 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:56.841457 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:56.842392 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:56.842605 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:56.842712 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:56.844918 (+ 2206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:41:56.844922 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:41:56.844924 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:41:56.844958 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:41:56.846559 (+ 1601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:41:56.847017 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:41:56.847022 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:41:56.847024 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:41:56.847101 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:41:56.847382 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:41:56.847385 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:41:56.847387 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:41:56.847550 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:41:56.847655 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 18:41:56.848063 (+ 408us) server_negotiation.cc:299] Negotiation successful
0929 18:41:56.848206 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260929 18:41:56.849738 3740 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.100.61" port: 41479 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 18:41:56 dist-test-slave-j1xs krb5kdc[3508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707316, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.100.62@KRBTEST.COM
Sep 29 18:41:56 dist-test-slave-j1xs krb5kdc[3508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707316, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.60@KRBTEST.COM for kudu/127.13.100.61@KRBTEST.COM
I20260929 18:41:56.857735 3593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:56.848359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42224 (local address 127.13.100.62:37443)
0929 18:41:56.848543 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0929 18:41:56.848548 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:41:56.848562 (+ 14us) server_negotiation.cc:407] Connection header received
0929 18:41:56.848678 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:41:56.848681 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:41:56.848724 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:41:56.848821 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:41:56.849442 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:56.850014 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:56.850906 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:56.851136 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:56.851252 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:56.853791 (+ 2539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:41:56.853795 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:41:56.853797 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:41:56.853821 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:41:56.855787 (+ 1966us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:41:56.856285 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:41:56.856288 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:41:56.856290 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:41:56.856385 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:41:56.856672 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:41:56.856678 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:41:56.856681 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:41:56.856946 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:41:56.857190 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0929 18:41:56.857345 (+ 155us) server_negotiation.cc:299] Negotiation successful
0929 18:41:56.857456 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":106,"threads_started":1}
I20260929 18:41:56.858698 3744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:56.850231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.61:41479 (local address 127.0.0.1:41486)
0929 18:41:56.850316 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:56.850330 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:41:56.850398 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:41:56.850713 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:41:56.850715 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:41:56.850873 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:41:56.850952 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:41:56.850957 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:41:56.851652 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:41:56.851658 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:41:56.852156 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:41:56.852163 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:41:56.852640 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:41:56.852643 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:41:56.852744 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:56.853293 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:41:56.853309 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:41:56.854824 (+ 1515us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:41:56.856886 (+ 2062us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:41:56.856889 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:41:56.856890 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:41:56.857070 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:41:56.857372 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:41:56.857377 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:41:56.857379 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:41:56.857474 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:41:56.858234 (+ 760us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:41:56.858237 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:41:56.858322 (+ 85us) client_negotiation.cc:769] Sending connection context
0929 18:41:56.858455 (+ 133us) client_negotiation.cc:240] Negotiation successful
0929 18:41:56.858584 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260929 18:41:56.858788 3670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:56.850370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41486 (local address 127.13.100.61:41479)
0929 18:41:56.850523 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0929 18:41:56.850527 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:41:56.850542 (+ 15us) server_negotiation.cc:407] Connection header received
0929 18:41:56.850581 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:41:56.850583 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:41:56.850617 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:41:56.850701 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:41:56.851060 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:56.851550 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:56.852276 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:56.852531 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:56.852641 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:56.854977 (+ 2336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:41:56.854982 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:41:56.854984 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:41:56.855018 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:41:56.856785 (+ 1767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:41:56.857177 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:41:56.857189 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:41:56.857191 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:41:56.857278 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:41:56.857574 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:41:56.857578 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:41:56.857579 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:41:56.857816 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:41:56.858271 (+ 455us) server_negotiation.cc:1035] Waiting for connection context
0929 18:41:56.858483 (+ 212us) server_negotiation.cc:299] Negotiation successful
0929 18:41:56.858638 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 18:41:56.859580 3664 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.100.60" port: 34553 } attempt: 3
I20260929 18:41:56.859812 3740 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.100.60" port: 34553 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 18:41:56 dist-test-slave-j1xs krb5kdc[3508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707316, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.100.61@KRBTEST.COM
Sep 29 18:41:56 dist-test-slave-j1xs krb5kdc[3508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707316, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.61@KRBTEST.COM for kudu/127.13.100.60@KRBTEST.COM
Sep 29 18:41:56 dist-test-slave-j1xs krb5kdc[3508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707316, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.60@KRBTEST.COM for kudu/127.13.100.60@KRBTEST.COM
I20260929 18:41:56.868815 3670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:56.859216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41502 (local address 127.13.100.61:41479)
0929 18:41:56.859333 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0929 18:41:56.859336 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:41:56.859349 (+ 13us) server_negotiation.cc:407] Connection header received
0929 18:41:56.859405 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:41:56.859411 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:41:56.859460 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:41:56.859533 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:41:56.860313 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:56.861014 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:56.862178 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:56.862441 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:56.862513 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:56.865299 (+ 2786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:41:56.865303 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:41:56.865305 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:41:56.865333 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:41:56.867058 (+ 1725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:41:56.867537 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:41:56.867543 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:41:56.867548 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:41:56.867636 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:41:56.867906 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:41:56.867912 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:41:56.867917 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:41:56.868147 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:41:56.868385 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0929 18:41:56.868523 (+ 138us) server_negotiation.cc:299] Negotiation successful
0929 18:41:56.868634 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 18:41:56.869643 3667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:56.859955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.60:34553 (local address 127.0.0.1:37148)
0929 18:41:56.860122 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:56.860135 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:41:56.860202 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:41:56.860468 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:41:56.860471 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:41:56.860657 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:41:56.860753 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:41:56.860766 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:41:56.862187 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:41:56.862190 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:41:56.862795 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:41:56.862803 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:41:56.863659 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:41:56.863664 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:41:56.863782 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:56.864259 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:41:56.864273 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:41:56.865739 (+ 1466us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:41:56.868075 (+ 2336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:41:56.868078 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:41:56.868080 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:41:56.868365 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:41:56.868686 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:41:56.868689 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:41:56.868692 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:41:56.868782 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:41:56.869229 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:41:56.869232 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:41:56.869320 (+ 88us) client_negotiation.cc:769] Sending connection context
0929 18:41:56.869423 (+ 103us) client_negotiation.cc:240] Negotiation successful
0929 18:41:56.869541 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"mutex_wait_us":1}
I20260929 18:41:56.870328 3743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:56.860020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37148 (local address 127.13.100.60:34553)
0929 18:41:56.860151 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0929 18:41:56.860155 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:41:56.860170 (+ 15us) server_negotiation.cc:407] Connection header received
0929 18:41:56.860299 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:41:56.860302 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:41:56.860338 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:41:56.860448 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:41:56.860870 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:56.862077 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:56.863173 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:56.863505 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:56.863645 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:56.865859 (+ 2214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:41:56.865865 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:41:56.865870 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:41:56.865915 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:41:56.867947 (+ 2032us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:41:56.868493 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:41:56.868497 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:41:56.868498 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:41:56.868571 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:41:56.868893 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:41:56.868896 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:41:56.868898 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:41:56.869108 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:41:56.869240 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0929 18:41:56.870065 (+ 825us) server_negotiation.cc:299] Negotiation successful
0929 18:41:56.870168 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 18:41:56.870460 3744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:56.860520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.60:34553 (local address 127.0.0.1:37158)
0929 18:41:56.860958 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:56.860971 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:41:56.861035 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:41:56.861358 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:41:56.861361 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:41:56.861517 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:41:56.861572 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:41:56.861577 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:41:56.862458 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:41:56.862463 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:41:56.863237 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:41:56.863249 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:41:56.863882 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:41:56.863885 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:41:56.863976 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:56.865390 (+ 1414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:41:56.865400 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:41:56.866852 (+ 1452us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:41:56.868985 (+ 2133us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:41:56.868988 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:41:56.868989 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:41:56.869200 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:41:56.869449 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:41:56.869454 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:41:56.869456 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:41:56.869533 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:41:56.869957 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:41:56.869960 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:41:56.870035 (+ 75us) client_negotiation.cc:769] Sending connection context
0929 18:41:56.870142 (+ 107us) client_negotiation.cc:240] Negotiation successful
0929 18:41:56.870266 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":353}
I20260929 18:41:56.870679 3746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:56.860753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37158 (local address 127.13.100.60:34553)
0929 18:41:56.861186 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0929 18:41:56.861192 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:41:56.861201 (+ 9us) server_negotiation.cc:407] Connection header received
0929 18:41:56.861225 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:41:56.861227 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:41:56.861255 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:41:56.861350 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:41:56.861676 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:56.862345 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:56.863363 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:56.863762 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:56.863873 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:56.866971 (+ 3098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:41:56.866974 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:41:56.866977 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:41:56.867001 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:41:56.868893 (+ 1892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:41:56.869289 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:41:56.869292 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:41:56.869294 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:41:56.869362 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:41:56.869627 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:41:56.869630 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:41:56.869632 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:41:56.869822 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:41:56.869938 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0929 18:41:56.870423 (+ 485us) server_negotiation.cc:299] Negotiation successful
0929 18:41:56.870521 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":105,"threads_started":1}
I20260929 18:41:56.871624 3589 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.100.60" port: 34553 } attempt: 3
I20260929 18:41:56.872834 13712 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:41:56.873888 3740 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 852636c4803e4dbca073acbb031b783c: Bootstrap starting.
I20260929 18:41:56.874034 3664 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a4b65aa97e14df392a22dff75dca5ac: Bootstrap starting.
I20260929 18:41:56.874292 3563 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "96303e1b5ab849f18213c81e071494b1"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42224
I20260929 18:41:56.876196 3664 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1a4b65aa97e14df392a22dff75dca5ac: Neither blocks nor log segments found. Creating new log.
I20260929 18:41:56.876332 3740 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 852636c4803e4dbca073acbb031b783c: Neither blocks nor log segments found. Creating new log.
I20260929 18:41:56.876957 3664 log.cc:824] T 00000000000000000000000000000000 P 1a4b65aa97e14df392a22dff75dca5ac: Log is configured to *not* fsync() on all Append() calls
I20260929 18:41:56.877069 3740 log.cc:824] T 00000000000000000000000000000000 P 852636c4803e4dbca073acbb031b783c: Log is configured to *not* fsync() on all Append() calls
I20260929 18:41:56.879114 3740 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 852636c4803e4dbca073acbb031b783c: No bootstrap required, opened a new log
I20260929 18:41:56.879135 3664 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a4b65aa97e14df392a22dff75dca5ac: No bootstrap required, opened a new log
Sep 29 18:41:56 dist-test-slave-j1xs krb5kdc[3508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707316, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.62@KRBTEST.COM for kudu/127.13.100.60@KRBTEST.COM
I20260929 18:41:56.883029 3740 raft_consensus.cc:348] T 00000000000000000000000000000000 P 852636c4803e4dbca073acbb031b783c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96303e1b5ab849f18213c81e071494b1" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 37443 } } peers { permanent_uuid: "1a4b65aa97e14df392a22dff75dca5ac" member_type: VOTER last_known_addr { host: "127.13.100.61" port: 41479 } } peers { permanent_uuid: "852636c4803e4dbca073acbb031b783c" member_type: VOTER last_known_addr { host: "127.13.100.60" port: 34553 } }
I20260929 18:41:56.883332 3740 raft_consensus.cc:729] T 00000000000000000000000000000000 P 852636c4803e4dbca073acbb031b783c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 852636c4803e4dbca073acbb031b783c, State: Initialized, Role: FOLLOWER
I20260929 18:41:56.883299 3664 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1a4b65aa97e14df392a22dff75dca5ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96303e1b5ab849f18213c81e071494b1" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 37443 } } peers { permanent_uuid: "1a4b65aa97e14df392a22dff75dca5ac" member_type: VOTER last_known_addr { host: "127.13.100.61" port: 41479 } } peers { permanent_uuid: "852636c4803e4dbca073acbb031b783c" member_type: VOTER last_known_addr { host: "127.13.100.60" port: 34553 } }
I20260929 18:41:56.883553 3664 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1a4b65aa97e14df392a22dff75dca5ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a4b65aa97e14df392a22dff75dca5ac, State: Initialized, Role: FOLLOWER
I20260929 18:41:56.883711 3740 consensus_queue.cc:260] T 00000000000000000000000000000000 P 852636c4803e4dbca073acbb031b783c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96303e1b5ab849f18213c81e071494b1" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 37443 } } peers { permanent_uuid: "1a4b65aa97e14df392a22dff75dca5ac" member_type: VOTER last_known_addr { host: "127.13.100.61" port: 41479 } } peers { permanent_uuid: "852636c4803e4dbca073acbb031b783c" member_type: VOTER last_known_addr { host: "127.13.100.60" port: 34553 } }
I20260929 18:41:56.883900 3664 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1a4b65aa97e14df392a22dff75dca5ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96303e1b5ab849f18213c81e071494b1" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 37443 } } peers { permanent_uuid: "1a4b65aa97e14df392a22dff75dca5ac" member_type: VOTER last_known_addr { host: "127.13.100.61" port: 41479 } } peers { permanent_uuid: "852636c4803e4dbca073acbb031b783c" member_type: VOTER last_known_addr { host: "127.13.100.60" port: 34553 } }
I20260929 18:41:56.884268 3749 sys_catalog.cc:455] T 00000000000000000000000000000000 P 852636c4803e4dbca073acbb031b783c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96303e1b5ab849f18213c81e071494b1" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 37443 } } peers { permanent_uuid: "1a4b65aa97e14df392a22dff75dca5ac" member_type: VOTER last_known_addr { host: "127.13.100.61" port: 41479 } } peers { permanent_uuid: "852636c4803e4dbca073acbb031b783c" member_type: VOTER last_known_addr { host: "127.13.100.60" port: 34553 } } }
I20260929 18:41:56.884553 3749 sys_catalog.cc:458] T 00000000000000000000000000000000 P 852636c4803e4dbca073acbb031b783c [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:41:56.885026 3740 sys_catalog.cc:564] T 00000000000000000000000000000000 P 852636c4803e4dbca073acbb031b783c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:41:56.885315 3664 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1a4b65aa97e14df392a22dff75dca5ac [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:41:56.886668 3592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:56.872004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.60:34553 (local address 127.0.0.1:37174)
0929 18:41:56.872199 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:56.872214 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 18:41:56.872292 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:41:56.877859 (+ 5567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:41:56.877863 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:41:56.878099 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:41:56.878203 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:41:56.878211 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:41:56.879246 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:41:56.879250 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:41:56.880030 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:41:56.880041 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:41:56.880715 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:41:56.880718 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:41:56.880799 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:56.881334 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:41:56.881348 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:41:56.882868 (+ 1520us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:41:56.884918 (+ 2050us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:41:56.884921 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:41:56.884923 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:41:56.885163 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:41:56.885481 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:41:56.885484 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:41:56.885486 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:41:56.885574 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:41:56.886076 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:41:56.886080 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:41:56.886194 (+ 114us) client_negotiation.cc:769] Sending connection context
0929 18:41:56.886317 (+ 123us) client_negotiation.cc:240] Negotiation successful
0929 18:41:56.886468 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103}
I20260929 18:41:56.887238 3746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:56.877528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37174 (local address 127.13.100.60:34553)
0929 18:41:56.877634 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0929 18:41:56.877639 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:41:56.877653 (+ 14us) server_negotiation.cc:407] Connection header received
0929 18:41:56.877695 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:41:56.877700 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:41:56.877742 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:41:56.877824 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:41:56.878339 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:56.879111 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:56.880165 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:56.880492 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:56.880586 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:56.883032 (+ 2446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:41:56.883036 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:41:56.883038 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:41:56.883061 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:41:56.884777 (+ 1716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:41:56.885279 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:41:56.885282 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:41:56.885284 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:41:56.885369 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:41:56.885693 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:41:56.885696 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:41:56.885698 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:41:56.885928 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:41:56.886024 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 18:41:56.886940 (+ 916us) server_negotiation.cc:299] Negotiation successful
0929 18:41:56.887052 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 18:41:56.890259 3589 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96303e1b5ab849f18213c81e071494b1: Bootstrap starting.
W20260929 18:41:56.890401 3770 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 852636c4803e4dbca073acbb031b783c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:41:56.890455 3770 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:41:56.890792 3750 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a4b65aa97e14df392a22dff75dca5ac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96303e1b5ab849f18213c81e071494b1" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 37443 } } peers { permanent_uuid: "1a4b65aa97e14df392a22dff75dca5ac" member_type: VOTER last_known_addr { host: "127.13.100.61" port: 41479 } } peers { permanent_uuid: "852636c4803e4dbca073acbb031b783c" member_type: VOTER last_known_addr { host: "127.13.100.60" port: 34553 } } }
I20260929 18:41:56.890911 3750 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a4b65aa97e14df392a22dff75dca5ac [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:41:56.892706 3589 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 96303e1b5ab849f18213c81e071494b1: Neither blocks nor log segments found. Creating new log.
I20260929 18:41:56.889604 3596 master_runner.cc:186] Error getting master registration for 127.13.100.62:37443: OK, instance_id { permanent_uuid: "96303e1b5ab849f18213c81e071494b1" instance_seqno: 1790707316393763 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 18:41:56.892841 3596 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260929 18:41:56.893461 3772 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1a4b65aa97e14df392a22dff75dca5ac: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:41:56.893517 3772 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:41:56.893496 3589 log.cc:824] T 00000000000000000000000000000000 P 96303e1b5ab849f18213c81e071494b1: Log is configured to *not* fsync() on all Append() calls
I20260929 18:41:56.894415 3671 master_runner.cc:186] Error getting master registration for 127.13.100.62:37443: OK, instance_id { permanent_uuid: "96303e1b5ab849f18213c81e071494b1" instance_seqno: 1790707316393763 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 18:41:56.898275 3671 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 18:41:56.899957 3589 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96303e1b5ab849f18213c81e071494b1: No bootstrap required, opened a new log
I20260929 18:41:56.904563 3589 raft_consensus.cc:348] T 00000000000000000000000000000000 P 96303e1b5ab849f18213c81e071494b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96303e1b5ab849f18213c81e071494b1" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 37443 } } peers { permanent_uuid: "1a4b65aa97e14df392a22dff75dca5ac" member_type: VOTER last_known_addr { host: "127.13.100.61" port: 41479 } } peers { permanent_uuid: "852636c4803e4dbca073acbb031b783c" member_type: VOTER last_known_addr { host: "127.13.100.60" port: 34553 } }
I20260929 18:41:56.904886 3589 raft_consensus.cc:729] T 00000000000000000000000000000000 P 96303e1b5ab849f18213c81e071494b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96303e1b5ab849f18213c81e071494b1, State: Initialized, Role: FOLLOWER
I20260929 18:41:56.905383 3589 consensus_queue.cc:260] T 00000000000000000000000000000000 P 96303e1b5ab849f18213c81e071494b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96303e1b5ab849f18213c81e071494b1" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 37443 } } peers { permanent_uuid: "1a4b65aa97e14df392a22dff75dca5ac" member_type: VOTER last_known_addr { host: "127.13.100.61" port: 41479 } } peers { permanent_uuid: "852636c4803e4dbca073acbb031b783c" member_type: VOTER last_known_addr { host: "127.13.100.60" port: 34553 } }
I20260929 18:41:56.905998 3776 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96303e1b5ab849f18213c81e071494b1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96303e1b5ab849f18213c81e071494b1" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 37443 } } peers { permanent_uuid: "1a4b65aa97e14df392a22dff75dca5ac" member_type: VOTER last_known_addr { host: "127.13.100.61" port: 41479 } } peers { permanent_uuid: "852636c4803e4dbca073acbb031b783c" member_type: VOTER last_known_addr { host: "127.13.100.60" port: 34553 } } }
I20260929 18:41:56.906179 3776 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96303e1b5ab849f18213c81e071494b1 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:41:56.907346 3563 raft_consensus.cc:482] T 00000000000000000000000000000000 P 96303e1b5ab849f18213c81e071494b1 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 18:41:56.907464 3563 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 96303e1b5ab849f18213c81e071494b1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:41:56.908679 3563 raft_consensus.cc:504] T 00000000000000000000000000000000 P 96303e1b5ab849f18213c81e071494b1 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96303e1b5ab849f18213c81e071494b1" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 37443 } } peers { permanent_uuid: "1a4b65aa97e14df392a22dff75dca5ac" member_type: VOTER last_known_addr { host: "127.13.100.61" port: 41479 } } peers { permanent_uuid: "852636c4803e4dbca073acbb031b783c" member_type: VOTER last_known_addr { host: "127.13.100.60" port: 34553 } }
I20260929 18:41:56.908877 3589 sys_catalog.cc:564] T 00000000000000000000000000000000 P 96303e1b5ab849f18213c81e071494b1 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:41:56.909567 3563 leader_election.cc:288] T 00000000000000000000000000000000 P 96303e1b5ab849f18213c81e071494b1 [CANDIDATE]: Term 1 election: Requested vote from peers 1a4b65aa97e14df392a22dff75dca5ac (127.13.100.61:41479), 852636c4803e4dbca073acbb031b783c (127.13.100.60:34553)
I20260929 18:41:56.910781 3638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "96303e1b5ab849f18213c81e071494b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "1a4b65aa97e14df392a22dff75dca5ac"
I20260929 18:41:56.910995 3638 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1a4b65aa97e14df392a22dff75dca5ac [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:41:56.912230 3638 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1a4b65aa97e14df392a22dff75dca5ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96303e1b5ab849f18213c81e071494b1 in term 1.
I20260929 18:41:56.913142 3714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "96303e1b5ab849f18213c81e071494b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "852636c4803e4dbca073acbb031b783c"
I20260929 18:41:56.913362 3714 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 852636c4803e4dbca073acbb031b783c [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:41:56.913584 3539 leader_election.cc:302] T 00000000000000000000000000000000 P 96303e1b5ab849f18213c81e071494b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a4b65aa97e14df392a22dff75dca5ac, 96303e1b5ab849f18213c81e071494b1; no voters:
I20260929 18:41:56.913836 3776 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 96303e1b5ab849f18213c81e071494b1 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:41:56.914033 3776 raft_consensus.cc:686] T 00000000000000000000000000000000 P 96303e1b5ab849f18213c81e071494b1 [term 1 LEADER]: Becoming Leader. State: Replica: 96303e1b5ab849f18213c81e071494b1, State: Running, Role: LEADER
I20260929 18:41:56.914341 3776 consensus_queue.cc:237] T 00000000000000000000000000000000 P 96303e1b5ab849f18213c81e071494b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96303e1b5ab849f18213c81e071494b1" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 37443 } } peers { permanent_uuid: "1a4b65aa97e14df392a22dff75dca5ac" member_type: VOTER last_known_addr { host: "127.13.100.61" port: 41479 } } peers { permanent_uuid: "852636c4803e4dbca073acbb031b783c" member_type: VOTER last_known_addr { host: "127.13.100.60" port: 34553 } }
I20260929 18:41:56.914695 3714 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 852636c4803e4dbca073acbb031b783c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96303e1b5ab849f18213c81e071494b1 in term 1.
I20260929 18:41:56.915942 3776 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96303e1b5ab849f18213c81e071494b1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 96303e1b5ab849f18213c81e071494b1. Latest consensus state: current_term: 1 leader_uuid: "96303e1b5ab849f18213c81e071494b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96303e1b5ab849f18213c81e071494b1" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 37443 } } peers { permanent_uuid: "1a4b65aa97e14df392a22dff75dca5ac" member_type: VOTER last_known_addr { host: "127.13.100.61" port: 41479 } } peers { permanent_uuid: "852636c4803e4dbca073acbb031b783c" member_type: VOTER last_known_addr { host: "127.13.100.60" port: 34553 } } }
I20260929 18:41:56.916038 3776 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96303e1b5ab849f18213c81e071494b1 [sys.catalog]: This master's current role is: LEADER
W20260929 18:41:56.919029 3787 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 96303e1b5ab849f18213c81e071494b1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:41:56.919101 3787 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:41:56.920629 3521 master_runner.cc:263] No leader master found from master 96303e1b5ab849f18213c81e071494b1
I20260929 18:41:56.920686 3521 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 18:41:56.920961 3794 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:41:56.921780 3794 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:41:56.934870 3746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:56.918959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37182 (local address 127.13.100.60:34553)
0929 18:41:56.919063 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0929 18:41:56.919068 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:41:56.926135 (+ 7067us) server_negotiation.cc:407] Connection header received
0929 18:41:56.926303 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:41:56.926306 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:41:56.926350 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:41:56.926409 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:41:56.927349 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:56.928104 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:56.929230 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:56.929570 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:56.929637 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:56.931305 (+ 1668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:41:56.931309 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:41:56.931312 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:41:56.931336 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:41:56.933114 (+ 1778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:41:56.933580 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:41:56.933584 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:41:56.933585 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:41:56.933662 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:41:56.933960 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:41:56.933964 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:41:56.933965 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:41:56.934169 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:41:56.934278 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 18:41:56.934602 (+ 324us) server_negotiation.cc:299] Negotiation successful
0929 18:41:56.934709 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 18:41:56.943894 3670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:56.918308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41516 (local address 127.13.100.61:41479)
0929 18:41:56.918420 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0929 18:41:56.918425 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:41:56.922113 (+ 3688us) server_negotiation.cc:407] Connection header received
0929 18:41:56.922297 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:41:56.922301 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:41:56.922345 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:41:56.922392 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:41:56.925179 (+ 2787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:56.925974 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:56.927165 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:56.927500 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:56.927571 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:56.938712 (+ 11141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:41:56.938716 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:41:56.938719 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:41:56.938747 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:41:56.940403 (+ 1656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:41:56.941612 (+ 1209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:41:56.941616 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:41:56.941617 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:41:56.941701 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:41:56.942005 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:41:56.942008 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:41:56.942010 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:41:56.942229 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:41:56.942323 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0929 18:41:56.943625 (+ 1302us) server_negotiation.cc:299] Negotiation successful
0929 18:41:56.943740 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 18:41:56.951803 3593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:56.919473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42234 (local address 127.13.100.62:37443)
0929 18:41:56.919568 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0929 18:41:56.919572 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:41:56.929719 (+ 10147us) server_negotiation.cc:407] Connection header received
0929 18:41:56.936472 (+ 6753us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:41:56.936476 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:41:56.936526 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:41:56.936578 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:41:56.937330 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:56.938087 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:56.942609 (+ 4522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:56.942941 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:56.943008 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:56.948312 (+ 5304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:41:56.948316 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:41:56.948318 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:41:56.948343 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:41:56.950054 (+ 1711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:41:56.950523 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:41:56.950527 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:41:56.950529 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:41:56.950609 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:41:56.950876 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:41:56.950879 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:41:56.950881 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:41:56.951104 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:41:56.951225 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0929 18:41:56.951544 (+ 319us) server_negotiation.cc:299] Negotiation successful
0929 18:41:56.951655 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260929 18:41:56.953961 3714 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 852636c4803e4dbca073acbb031b783c [term 1 FOLLOWER]: Refusing update from remote peer 96303e1b5ab849f18213c81e071494b1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:41:56.954699 3638 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1a4b65aa97e14df392a22dff75dca5ac [term 1 FOLLOWER]: Refusing update from remote peer 96303e1b5ab849f18213c81e071494b1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:41:56.955413 3792 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 96303e1b5ab849f18213c81e071494b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a4b65aa97e14df392a22dff75dca5ac" member_type: VOTER last_known_addr { host: "127.13.100.61" port: 41479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:41:56.955578 3776 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 96303e1b5ab849f18213c81e071494b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "852636c4803e4dbca073acbb031b783c" member_type: VOTER last_known_addr { host: "127.13.100.60" port: 34553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:41:56.959510 3749 sys_catalog.cc:455] T 00000000000000000000000000000000 P 852636c4803e4dbca073acbb031b783c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 96303e1b5ab849f18213c81e071494b1. Latest consensus state: current_term: 1 leader_uuid: "96303e1b5ab849f18213c81e071494b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96303e1b5ab849f18213c81e071494b1" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 37443 } } peers { permanent_uuid: "1a4b65aa97e14df392a22dff75dca5ac" member_type: VOTER last_known_addr { host: "127.13.100.61" port: 41479 } } peers { permanent_uuid: "852636c4803e4dbca073acbb031b783c" member_type: VOTER last_known_addr { host: "127.13.100.60" port: 34553 } } }
I20260929 18:41:56.959627 3749 sys_catalog.cc:458] T 00000000000000000000000000000000 P 852636c4803e4dbca073acbb031b783c [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:41:56.959673 3750 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a4b65aa97e14df392a22dff75dca5ac [sys.catalog]: SysCatalogTable state changed. Reason: New leader 96303e1b5ab849f18213c81e071494b1. Latest consensus state: current_term: 1 leader_uuid: "96303e1b5ab849f18213c81e071494b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96303e1b5ab849f18213c81e071494b1" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 37443 } } peers { permanent_uuid: "1a4b65aa97e14df392a22dff75dca5ac" member_type: VOTER last_known_addr { host: "127.13.100.61" port: 41479 } } peers { permanent_uuid: "852636c4803e4dbca073acbb031b783c" member_type: VOTER last_known_addr { host: "127.13.100.60" port: 34553 } } }
I20260929 18:41:56.959785 3750 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a4b65aa97e14df392a22dff75dca5ac [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:41:56.966959 3750 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a4b65aa97e14df392a22dff75dca5ac [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "96303e1b5ab849f18213c81e071494b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96303e1b5ab849f18213c81e071494b1" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 37443 } } peers { permanent_uuid: "1a4b65aa97e14df392a22dff75dca5ac" member_type: VOTER last_known_addr { host: "127.13.100.61" port: 41479 } } peers { permanent_uuid: "852636c4803e4dbca073acbb031b783c" member_type: VOTER last_known_addr { host: "127.13.100.60" port: 34553 } } }
I20260929 18:41:56.967097 3750 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a4b65aa97e14df392a22dff75dca5ac [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:41:56.968142 3749 sys_catalog.cc:455] T 00000000000000000000000000000000 P 852636c4803e4dbca073acbb031b783c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "96303e1b5ab849f18213c81e071494b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96303e1b5ab849f18213c81e071494b1" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 37443 } } peers { permanent_uuid: "1a4b65aa97e14df392a22dff75dca5ac" member_type: VOTER last_known_addr { host: "127.13.100.61" port: 41479 } } peers { permanent_uuid: "852636c4803e4dbca073acbb031b783c" member_type: VOTER last_known_addr { host: "127.13.100.60" port: 34553 } } }
I20260929 18:41:56.968251 3749 sys_catalog.cc:458] T 00000000000000000000000000000000 P 852636c4803e4dbca073acbb031b783c [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:41:56.968734 3801 mvcc.cc:205] Tried to move back new op lower bound from 7334737170216783872 to 7334737170084196352. Current Snapshot: MvccSnapshot[applied={T|T < 7334737170216783872}]
I20260929 18:41:56.969532 3792 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96303e1b5ab849f18213c81e071494b1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "96303e1b5ab849f18213c81e071494b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96303e1b5ab849f18213c81e071494b1" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 37443 } } peers { permanent_uuid: "1a4b65aa97e14df392a22dff75dca5ac" member_type: VOTER last_known_addr { host: "127.13.100.61" port: 41479 } } peers { permanent_uuid: "852636c4803e4dbca073acbb031b783c" member_type: VOTER last_known_addr { host: "127.13.100.60" port: 34553 } } }
I20260929 18:41:56.969642 3792 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96303e1b5ab849f18213c81e071494b1 [sys.catalog]: This master's current role is: LEADER
I20260929 18:41:56.969723 3792 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96303e1b5ab849f18213c81e071494b1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "96303e1b5ab849f18213c81e071494b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96303e1b5ab849f18213c81e071494b1" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 37443 } } peers { permanent_uuid: "1a4b65aa97e14df392a22dff75dca5ac" member_type: VOTER last_known_addr { host: "127.13.100.61" port: 41479 } } peers { permanent_uuid: "852636c4803e4dbca073acbb031b783c" member_type: VOTER last_known_addr { host: "127.13.100.60" port: 34553 } } }
I20260929 18:41:56.969779 3792 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96303e1b5ab849f18213c81e071494b1 [sys.catalog]: This master's current role is: LEADER
I20260929 18:41:56.970749 3799 mvcc.cc:205] Tried to move back new op lower bound from 7334737170216783872 to 7334737170084196352. Current Snapshot: MvccSnapshot[applied={T|T < 7334737170216783872}]
I20260929 18:41:56.971053 3800 mvcc.cc:205] Tried to move back new op lower bound from 7334737170216783872 to 7334737170084196352. Current Snapshot: MvccSnapshot[applied={T|T < 7334737170216783872}]
I20260929 18:41:56.980580 3794 catalog_manager.cc:1347] Generated new cluster ID: df7b8020466744a98716c6a9727cfa4c
I20260929 18:41:56.980667 3794 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:41:57.007005 3794 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:41:57.008322 3794 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:41:57.023101 3794 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 96303e1b5ab849f18213c81e071494b1: Generated new TSK 0
I20260929 18:41:57.024024 3794 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 18:41:57.050959 3593 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:41:57.048078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42248 (local address 127.13.100.62:37443)
0929 18:41:57.048231 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0929 18:41:57.048238 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 18:41:57.049831 (+ 1593us) server_negotiation.cc:407] Connection header received
0929 18:41:57.050079 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:41:57.050082 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:41:57.050130 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:41:57.050188 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:41:57.050820 (+ 632us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42248: BlockingRecv error: recv got EOF from 127.0.0.1:42248 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260929 18:41:57.052956 3670 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:41:57.048899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41530 (local address 127.13.100.61:41479)
0929 18:41:57.049011 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0929 18:41:57.049016 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:41:57.051575 (+ 2559us) server_negotiation.cc:407] Connection header received
0929 18:41:57.052145 (+ 570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:41:57.052148 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:41:57.052198 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:41:57.052251 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:41:57.052846 (+ 595us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41530: BlockingRecv error: recv got EOF from 127.0.0.1:41530 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260929 18:41:57.054221 3746 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:41:57.049543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37184 (local address 127.13.100.60:34553)
0929 18:41:57.049681 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0929 18:41:57.049685 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:41:57.052046 (+ 2361us) server_negotiation.cc:407] Connection header received
0929 18:41:57.053447 (+ 1401us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:41:57.053451 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:41:57.053500 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:41:57.053553 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:41:57.054097 (+ 544us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37184: BlockingRecv error: recv got EOF from 127.0.0.1:37184 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260929 18:41:57.058964 3670 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:41:57.057379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41538 (local address 127.13.100.61:41479)
0929 18:41:57.057486 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0929 18:41:57.057491 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:41:57.058014 (+ 523us) server_negotiation.cc:407] Connection header received
0929 18:41:57.058243 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:41:57.058246 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:41:57.058322 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:41:57.058376 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:41:57.058862 (+ 486us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41538: BlockingRecv error: recv got EOF from 127.0.0.1:41538 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260929 18:41:57.064009 3670 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:41:57.062425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41544 (local address 127.13.100.61:41479)
0929 18:41:57.062526 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0929 18:41:57.062529 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:41:57.063090 (+ 561us) server_negotiation.cc:407] Connection header received
0929 18:41:57.063304 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:41:57.063307 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:41:57.063356 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:41:57.063408 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:41:57.063914 (+ 506us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41544: BlockingRecv error: recv got EOF from 127.0.0.1:41544 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260929 18:41:57.069744 3670 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:41:57.068312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41556 (local address 127.13.100.61:41479)
0929 18:41:57.068408 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0929 18:41:57.068411 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:41:57.068881 (+ 470us) server_negotiation.cc:407] Connection header received
0929 18:41:57.069052 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:41:57.069055 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:41:57.069117 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:41:57.069168 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:41:57.069649 (+ 481us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41556: BlockingRecv error: recv got EOF from 127.0.0.1:41556 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260929 18:41:57.077566 3670 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:41:57.075982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41558 (local address 127.13.100.61:41479)
0929 18:41:57.076120 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0929 18:41:57.076124 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:41:57.076658 (+ 534us) server_negotiation.cc:407] Connection header received
0929 18:41:57.076849 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:41:57.076852 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:41:57.076901 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:41:57.076953 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:41:57.077467 (+ 514us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41558: BlockingRecv error: recv got EOF from 127.0.0.1:41558 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260929 18:41:57.089489 3670 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:41:57.087929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41574 (local address 127.13.100.61:41479)
0929 18:41:57.088031 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0929 18:41:57.088036 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:41:57.088579 (+ 543us) server_negotiation.cc:407] Connection header received
0929 18:41:57.088770 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:41:57.088774 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:41:57.088827 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:41:57.088878 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:41:57.089375 (+ 497us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41574: BlockingRecv error: recv got EOF from 127.0.0.1:41574 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260929 18:41:57.109181 3670 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:41:57.107974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41578 (local address 127.13.100.61:41479)
0929 18:41:57.108126 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0929 18:41:57.108131 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:41:57.108298 (+ 167us) server_negotiation.cc:407] Connection header received
0929 18:41:57.108434 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:41:57.108437 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:41:57.108481 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:41:57.108550 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:41:57.109024 (+ 474us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41578: BlockingRecv error: recv got EOF from 127.0.0.1:41578 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260929 18:41:57.144460 3670 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:41:57.143237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41590 (local address 127.13.100.61:41479)
0929 18:41:57.143406 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0929 18:41:57.143411 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:41:57.143539 (+ 128us) server_negotiation.cc:407] Connection header received
0929 18:41:57.143713 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:41:57.143717 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:41:57.143776 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:41:57.143873 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:41:57.144324 (+ 451us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41590: BlockingRecv error: recv got EOF from 127.0.0.1:41590 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260929 18:41:57.211910 3670 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:41:57.210685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41594 (local address 127.13.100.61:41479)
0929 18:41:57.210864 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 18:41:57.210869 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:41:57.211027 (+ 158us) server_negotiation.cc:407] Connection header received
0929 18:41:57.211198 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:41:57.211201 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:41:57.211255 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:41:57.211343 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:41:57.211794 (+ 451us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41594: BlockingRecv error: recv got EOF from 127.0.0.1:41594 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260929 18:41:57.343293 3670 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:41:57.342010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41602 (local address 127.13.100.61:41479)
0929 18:41:57.342204 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0929 18:41:57.342209 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:41:57.342368 (+ 159us) server_negotiation.cc:407] Connection header received
0929 18:41:57.342557 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:41:57.342561 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:41:57.342618 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:41:57.342710 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:41:57.343176 (+ 466us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41602: BlockingRecv error: recv got EOF from 127.0.0.1:41602 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
W20260929 18:41:57.602716 3670 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:41:57.601496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41614 (local address 127.13.100.61:41479)
0929 18:41:57.601622 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0929 18:41:57.601626 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:41:57.601671 (+ 45us) server_negotiation.cc:407] Connection header received
0929 18:41:57.601816 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:41:57.601818 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:41:57.601863 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:41:57.601960 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:41:57.602593 (+ 633us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41614: BlockingRecv error: recv got EOF from 127.0.0.1:41614 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 18:41:57.891657 3770 catalog_manager.cc:1259] Loaded cluster ID: df7b8020466744a98716c6a9727cfa4c
I20260929 18:41:57.891741 3770 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 852636c4803e4dbca073acbb031b783c: loading cluster ID for follower catalog manager: success
I20260929 18:41:57.893364 3770 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 852636c4803e4dbca073acbb031b783c: acquiring CA information for follower catalog manager: success
I20260929 18:41:57.894577 3772 catalog_manager.cc:1259] Loaded cluster ID: df7b8020466744a98716c6a9727cfa4c
I20260929 18:41:57.894636 3772 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1a4b65aa97e14df392a22dff75dca5ac: loading cluster ID for follower catalog manager: success
I20260929 18:41:57.894593 3770 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 852636c4803e4dbca073acbb031b783c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 18:41:57.896168 3772 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1a4b65aa97e14df392a22dff75dca5ac: acquiring CA information for follower catalog manager: success
I20260929 18:41:57.897373 3772 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1a4b65aa97e14df392a22dff75dca5ac: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 18:41:58.120995 3872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:58.116841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41616 (local address 127.13.100.61:41479)
0929 18:41:58.117122 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0929 18:41:58.117127 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:41:58.117209 (+ 82us) server_negotiation.cc:407] Connection header received
0929 18:41:58.117383 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:41:58.117389 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:41:58.117521 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:41:58.117643 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:41:58.117955 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:58.118581 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:58.119467 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:58.119702 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:58.119817 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:58.120149 (+ 332us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:41:58.120569 (+ 420us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:41:58.120729 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0929 18:41:58.120765 (+ 36us) server_negotiation.cc:299] Negotiation successful
0929 18:41:58.120821 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":113,"threads_started":1}
I20260929 18:41:58.128033 3878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:58.124215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37190 (local address 127.13.100.60:34553)
0929 18:41:58.124456 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0929 18:41:58.124462 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:41:58.124489 (+ 27us) server_negotiation.cc:407] Connection header received
0929 18:41:58.124577 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:41:58.124579 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:41:58.124682 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:41:58.124809 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:41:58.125037 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:58.125549 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:58.126470 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:58.126707 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:58.126821 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:58.127127 (+ 306us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:41:58.127642 (+ 515us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:41:58.127777 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0929 18:41:58.127817 (+ 40us) server_negotiation.cc:299] Negotiation successful
0929 18:41:58.127871 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":58,"threads_started":1}
I20260929 18:41:58.135548 3872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:58.131433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41626 (local address 127.13.100.61:41479)
0929 18:41:58.131574 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0929 18:41:58.131599 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0929 18:41:58.131931 (+ 332us) server_negotiation.cc:407] Connection header received
0929 18:41:58.132171 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:41:58.132174 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:41:58.132231 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:41:58.132283 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:41:58.132723 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:58.133245 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:58.134257 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:58.134622 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:58.134741 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:58.135001 (+ 260us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:41:58.135100 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:41:58.135168 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 18:41:58.135329 (+ 161us) server_negotiation.cc:299] Negotiation successful
0929 18:41:58.135388 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 18:41:58.135653 3878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:58.131825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37204 (local address 127.13.100.60:34553)
0929 18:41:58.131947 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0929 18:41:58.131952 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:41:58.131972 (+ 20us) server_negotiation.cc:407] Connection header received
0929 18:41:58.132189 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:41:58.132195 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:41:58.132246 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:41:58.132326 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:41:58.132563 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:58.133058 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:58.134333 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:58.134658 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:58.134756 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:58.135140 (+ 384us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:41:58.135222 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:41:58.135274 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0929 18:41:58.135457 (+ 183us) server_negotiation.cc:299] Negotiation successful
0929 18:41:58.135513 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 18:41:58.136701 3885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:58.131325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42260 (local address 127.13.100.62:37443)
0929 18:41:58.131587 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0929 18:41:58.131592 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:41:58.131829 (+ 237us) server_negotiation.cc:407] Connection header received
0929 18:41:58.132137 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:41:58.132144 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:41:58.132308 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:41:58.132434 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:41:58.133400 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:58.134226 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:58.135169 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:58.135474 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:58.135585 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:58.135909 (+ 324us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:41:58.136283 (+ 374us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:41:58.136390 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 18:41:58.136476 (+ 86us) server_negotiation.cc:299] Negotiation successful
0929 18:41:58.136535 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":63,"threads_started":1}
I20260929 18:41:58.144579 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 3521
I20260929 18:41:58.151079 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 3596
I20260929 18:41:58.156724 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 3671
2026-09-29T18:41:58Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3106 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707036041078-13712-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:41:58 dist-test-slave-j1xs krb5kdc[3889](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:41:58 dist-test-slave-j1xs krb5kdc[3889](info): set up 2 sockets
Sep 29 18:41:58 dist-test-slave-j1xs krb5kdc[3889](info): commencing operation
krb5kdc: starting...
W20260929 18:41:59.194234 13712 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707036041078-13712-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707036041078-13712-0/krb5kdc/test-user.keytab.
Sep 29 18:41:59 dist-test-slave-j1xs krb5kdc[3889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707319, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:41:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:41:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.100.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.100.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.100.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.62.keytab.
Entry for principal kudu/127.13.100.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.62.keytab.
I20260929 18:41:59.313951 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707036041078-13712-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707036041078-13712-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707036041078-13712-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.100.62:36405
--webserver_interface=127.13.100.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.62.keytab
--principal=kudu/127.13.100.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.100.20:41937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.100.62:36405,127.13.100.61:34345,127.13.100.60:44933
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707036041078-13712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707036041078-13712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707036041078-13712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:41:59.421597 3902 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 18:41:59.421988 3902 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:41:59.422056 3902 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:41:59.422142 3902 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:41:59.426128 3902 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:41:59.426221 3902 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:41:59.426262 3902 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:41:59.426299 3902 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:41:59.426331 3902 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:41:59.432552 3902 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.100.20:41937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707036041078-13712-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707036041078-13712-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.100.62:36405,127.13.100.61:34345,127.13.100.60:44933
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.62.keytab
--principal=kudu/127.13.100.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.100.62:36405
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.100.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3902
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707036041078-13712-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:41:59.433954 3902 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:41:59.435068 3902 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:41:59.443742 3908 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:41:59.443778 3910 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:41:59.443742 3907 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:41:59.443852 3902 server_base.cc:1034] running on GCE node
I20260929 18:41:59.444572 3902 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:41:59.445645 3902 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:41:59.446826 3902 hybrid_clock.cc:648] HybridClock initialized: now 1790707319446813 us; error 70 us; skew 500 ppm
Sep 29 18:41:59 dist-test-slave-j1xs krb5kdc[3889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707319, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:41:59.449762 3902 init.cc:377] Logged in from keytab as kudu/127.13.100.62@KRBTEST.COM (short username kudu)
I20260929 18:41:59.450836 3902 webserver.cc:460] Webserver started at http://127.13.100.62:38859/ using document root <none> and password file <none>
I20260929 18:41:59.451416 3902 fs_manager.cc:359] Metadata directory not provided
I20260929 18:41:59.451505 3902 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:41:59.451730 3902 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:41:59.453739 3902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707036041078-13712-0/minicluster-data/master-0/data/instance:
uuid: "667b13f021124ab7a5cf950563a3cd31"
format_stamp: "Formatted at 2026-09-29 18:41:59 on dist-test-slave-j1xs"
server_key: "0190e22ec6bb942ae9911ba847b0f9ef"
server_key_iv: "5511b9934545b4ed082f5b0df1a2b3d5"
server_key_version: "encryptionkey@0"
I20260929 18:41:59.454368 3902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707036041078-13712-0/minicluster-data/master-0/wal/instance:
uuid: "667b13f021124ab7a5cf950563a3cd31"
format_stamp: "Formatted at 2026-09-29 18:41:59 on dist-test-slave-j1xs"
server_key: "0190e22ec6bb942ae9911ba847b0f9ef"
server_key_iv: "5511b9934545b4ed082f5b0df1a2b3d5"
server_key_version: "encryptionkey@0"
I20260929 18:41:59.458132 3902 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 18:41:59.460547 3917 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:41:59.461747 3902 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:41:59.461928 3902 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707036041078-13712-0/minicluster-data/master-0/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707036041078-13712-0/minicluster-data/master-0/wal
uuid: "667b13f021124ab7a5cf950563a3cd31"
format_stamp: "Formatted at 2026-09-29 18:41:59 on dist-test-slave-j1xs"
server_key: "0190e22ec6bb942ae9911ba847b0f9ef"
server_key_iv: "5511b9934545b4ed082f5b0df1a2b3d5"
server_key_version: "encryptionkey@0"
I20260929 18:41:59.462049 3902 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707036041078-13712-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707036041078-13712-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707036041078-13712-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:41:59.477669 3902 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:41:59.480824 3902 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:41:59.481025 3902 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:41:59.487733 3902 rpc_server.cc:307] RPC server started. Bound to: 127.13.100.62:36405
I20260929 18:41:59.488003 3969 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.100.62:36405 every 8 connection(s)
I20260929 18:41:59.488734 3902 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707036041078-13712-0/minicluster-data/master-0/data/info.pb
I20260929 18:41:59.490120 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 3902
I20260929 18:41:59.490250 13712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707036041078-13712-0/minicluster-data/master-0/wal/instance
I20260929 18:41:59.490605 13712 external_mini_cluster.cc:1428] Setting key 2bbac804ec91be00c3bb31826d9ad3c5
I20260929 18:41:59.491894 3970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:41:59.497287 3970 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.100.62" port: 36405 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.13.100.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.100.61@KRBTEST.COM" created.
Sep 29 18:41:59 dist-test-slave-j1xs krb5kdc[3889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707319, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.62@KRBTEST.COM for kudu/127.13.100.62@KRBTEST.COM
I20260929 18:41:59.513099 3973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:59.500013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:36405 (local address 127.0.0.1:52960)
0929 18:41:59.500620 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:59.500666 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 18:41:59.501504 (+ 838us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:41:59.502224 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:41:59.502230 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:41:59.502655 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:41:59.502940 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:41:59.502973 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:41:59.504126 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:41:59.504130 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:41:59.504821 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:41:59.504829 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:41:59.505441 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:41:59.505445 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:41:59.505618 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:59.506258 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:41:59.506273 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:41:59.508063 (+ 1790us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:41:59.510059 (+ 1996us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:41:59.510069 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:41:59.510078 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:41:59.510297 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:41:59.510616 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:41:59.510619 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:41:59.510621 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:41:59.510749 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:41:59.511407 (+ 658us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:41:59.511413 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:41:59.511554 (+ 141us) client_negotiation.cc:769] Sending connection context
0929 18:41:59.511724 (+ 170us) client_negotiation.cc:240] Negotiation successful
0929 18:41:59.512107 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":358,"mutex_wait_us":134,"thread_start_us":131,"threads_started":1}
I20260929 18:41:59.513099 3974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:59.500013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52960 (local address 127.13.100.62:36405)
0929 18:41:59.500558 (+ 545us) server_negotiation.cc:206] Beginning negotiation
0929 18:41:59.500576 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 18:41:59.500765 (+ 189us) server_negotiation.cc:407] Connection header received
0929 18:41:59.501799 (+ 1034us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:41:59.501803 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:41:59.502046 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:41:59.502224 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:41:59.503128 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:59.503994 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:59.504936 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:59.505294 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:59.505456 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:59.508205 (+ 2749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:41:59.508216 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:41:59.508226 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:41:59.508245 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:41:59.509935 (+ 1690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:41:59.510402 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:41:59.510408 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:41:59.510409 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:41:59.510511 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:41:59.510847 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:41:59.510853 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:41:59.510855 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:41:59.511235 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:41:59.511434 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0929 18:41:59.511837 (+ 403us) server_negotiation.cc:299] Negotiation successful
0929 18:41:59.512052 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":48,"server-negotiator.queue_time_us":431,"spinlock_wait_cycles":28032,"thread_start_us":165,"threads_started":1}
I20260929 18:41:59.515563 3970 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.100.61" port: 34345 } has no permanent_uuid. Determining permanent_uuid...
I20260929 18:41:59.516361 3973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:59.516073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.61:34345 (local address 127.0.0.1:50050)
0929 18:41:59.516190 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:59.516287 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.61:34345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260929 18:41:59.516547 3921 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.100.61:34345: connect: Connection refused (error 111)
I20260929 18:41:59.517352 3973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:59.517091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.61:34345 (local address 127.0.0.1:50054)
0929 18:41:59.517244 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:59.517291 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.61:34345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
W20260929 18:41:59.517513 3970 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.100.61:34345: Network error: Client connection negotiation failed: client connection to 127.13.100.61:34345: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.100.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.61.keytab.
Entry for principal kudu/127.13.100.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.61.keytab.
I20260929 18:41:59.518875 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707036041078-13712-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707036041078-13712-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707036041078-13712-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707036041078-13712-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.100.61:34345
--webserver_interface=127.13.100.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.61.keytab
--principal=kudu/127.13.100.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.100.20:41937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.100.62:36405,127.13.100.61:34345,127.13.100.60:44933
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707036041078-13712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707036041078-13712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707036041078-13712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 18:41:59.566722 3970 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.100.61" port: 34345 } attempt: 1
I20260929 18:41:59.567660 3973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:59.567279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.61:34345 (local address 127.0.0.1:50068)
0929 18:41:59.567470 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:59.567552 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.61:34345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260929 18:41:59.568531 3973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:59.568255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.61:34345 (local address 127.0.0.1:50072)
0929 18:41:59.568386 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:59.568447 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.61:34345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260929 18:41:59.568717 3970 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.100.61:34345: Network error: Client connection negotiation failed: client connection to 127.13.100.61:34345: connect: Connection refused (error 111)
W20260929 18:41:59.628705 3977 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 18:41:59.629036 3977 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:41:59.629065 3977 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:41:59.629191 3977 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:41:59.633374 3977 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:41:59.633450 3977 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:41:59.633473 3977 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:41:59.633493 3977 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:41:59.633551 3977 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:41:59.636921 3970 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.100.61" port: 34345 } attempt: 2
I20260929 18:41:59.637821 3973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:59.637485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.61:34345 (local address 127.0.0.1:50088)
0929 18:41:59.637655 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:59.637730 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.61:34345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260929 18:41:59.638659 3973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:59.638431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.61:34345 (local address 127.0.0.1:50094)
0929 18:41:59.638538 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:59.638592 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.61:34345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260929 18:41:59.638835 3970 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.100.61:34345: Network error: Client connection negotiation failed: client connection to 127.13.100.61:34345: connect: Connection refused (error 111)
I20260929 18:41:59.639658 3977 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.100.20:41937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707036041078-13712-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707036041078-13712-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.13.100.62:36405,127.13.100.61:34345,127.13.100.60:44933
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.61.keytab
--principal=kudu/127.13.100.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.100.61:34345
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707036041078-13712-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.13.100.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3977
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707036041078-13712-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:41:59.640758 3977 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:41:59.641762 3977 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:41:59.650264 3985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:41:59.650328 3983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:41:59.650332 3982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:41:59.650478 3977 server_base.cc:1034] running on GCE node
I20260929 18:41:59.651178 3977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:41:59.652347 3977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:41:59.653565 3977 hybrid_clock.cc:648] HybridClock initialized: now 1790707319653527 us; error 51 us; skew 500 ppm
Sep 29 18:41:59 dist-test-slave-j1xs krb5kdc[3889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707319, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:41:59.656823 3977 init.cc:377] Logged in from keytab as kudu/127.13.100.61@KRBTEST.COM (short username kudu)
I20260929 18:41:59.657732 3977 webserver.cc:460] Webserver started at http://127.13.100.61:41655/ using document root <none> and password file <none>
I20260929 18:41:59.658360 3977 fs_manager.cc:359] Metadata directory not provided
I20260929 18:41:59.658463 3977 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:41:59.658692 3977 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:41:59.660719 3977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707036041078-13712-0/minicluster-data/master-1/data/instance:
uuid: "af2dec8be43940c7823a2d8268ddaf47"
format_stamp: "Formatted at 2026-09-29 18:41:59 on dist-test-slave-j1xs"
server_key: "5932fa3860a2a43565a51006a4f85393"
server_key_iv: "eb01953cff9261949c8dd372455fcde8"
server_key_version: "encryptionkey@0"
I20260929 18:41:59.661278 3977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707036041078-13712-0/minicluster-data/master-1/wal/instance:
uuid: "af2dec8be43940c7823a2d8268ddaf47"
format_stamp: "Formatted at 2026-09-29 18:41:59 on dist-test-slave-j1xs"
server_key: "5932fa3860a2a43565a51006a4f85393"
server_key_iv: "eb01953cff9261949c8dd372455fcde8"
server_key_version: "encryptionkey@0"
I20260929 18:41:59.665140 3977 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 18:41:59.667711 3992 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:41:59.668977 3977 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:41:59.669127 3977 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707036041078-13712-0/minicluster-data/master-1/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707036041078-13712-0/minicluster-data/master-1/wal
uuid: "af2dec8be43940c7823a2d8268ddaf47"
format_stamp: "Formatted at 2026-09-29 18:41:59 on dist-test-slave-j1xs"
server_key: "5932fa3860a2a43565a51006a4f85393"
server_key_iv: "eb01953cff9261949c8dd372455fcde8"
server_key_version: "encryptionkey@0"
I20260929 18:41:59.669250 3977 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707036041078-13712-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707036041078-13712-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707036041078-13712-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:41:59.680987 3977 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:41:59.684310 3977 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:41:59.684522 3977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:41:59.691241 3977 rpc_server.cc:307] RPC server started. Bound to: 127.13.100.61:34345
I20260929 18:41:59.691514 4044 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.100.61:34345 every 8 connection(s)
I20260929 18:41:59.692267 3977 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707036041078-13712-0/minicluster-data/master-1/data/info.pb
I20260929 18:41:59.694856 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 3977
I20260929 18:41:59.694998 13712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707036041078-13712-0/minicluster-data/master-1/wal/instance
I20260929 18:41:59.695109 4045 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:41:59.695310 13712 external_mini_cluster.cc:1428] Setting key 7318d0124a888e1f4f8f3a2c8ed279b9
I20260929 18:41:59.700002 4045 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.100.62" port: 36405 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.13.100.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.100.60@KRBTEST.COM" created.
Sep 29 18:41:59 dist-test-slave-j1xs krb5kdc[3889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707319, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.61@KRBTEST.COM for kudu/127.13.100.62@KRBTEST.COM
I20260929 18:41:59.714749 3974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:59.702510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52974 (local address 127.13.100.62:36405)
0929 18:41:59.702674 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0929 18:41:59.702679 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:41:59.703530 (+ 851us) server_negotiation.cc:407] Connection header received
0929 18:41:59.704500 (+ 970us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:41:59.704503 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:41:59.704554 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:41:59.704640 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:41:59.706093 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:59.706614 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:59.707929 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:59.708180 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:59.708303 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:59.710963 (+ 2660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:41:59.710966 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:41:59.710968 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:41:59.710994 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:41:59.712619 (+ 1625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:41:59.713153 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:41:59.713158 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:41:59.713160 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:41:59.713242 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:41:59.713597 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:41:59.713602 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:41:59.713604 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:41:59.713800 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:41:59.713938 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0929 18:41:59.714447 (+ 509us) server_negotiation.cc:299] Negotiation successful
0929 18:41:59.714573 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 18:41:59.715534 4048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:59.702888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:36405 (local address 127.0.0.1:52974)
0929 18:41:59.703354 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:59.703404 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 18:41:59.704281 (+ 877us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:41:59.704916 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:41:59.704923 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:41:59.705342 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:41:59.705918 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:41:59.705944 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:41:59.706766 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:41:59.706769 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:41:59.707786 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:41:59.707796 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:41:59.708341 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:41:59.708345 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:41:59.708517 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:59.709137 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:41:59.709153 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:41:59.710784 (+ 1631us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:41:59.712760 (+ 1976us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:41:59.712766 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:41:59.712781 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:41:59.713047 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:41:59.713335 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:41:59.713340 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:41:59.713341 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:41:59.713501 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:41:59.713937 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:41:59.713943 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:41:59.714188 (+ 245us) client_negotiation.cc:769] Sending connection context
0929 18:41:59.714372 (+ 184us) client_negotiation.cc:240] Negotiation successful
0929 18:41:59.714689 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":135,"threads_started":1}
I20260929 18:41:59.717175 4045 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.100.61" port: 34345 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 18:41:59 dist-test-slave-j1xs krb5kdc[3889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707319, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.61@KRBTEST.COM for kudu/127.13.100.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.100.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.60.keytab.
Entry for principal kudu/127.13.100.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.60.keytab.
I20260929 18:41:59.725021 13712 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
/tmp/dist-test-taskpp0tHW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707036041078-13712-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707036041078-13712-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707036041078-13712-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707036041078-13712-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.100.60:44933
--webserver_interface=127.13.100.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.60.keytab
--principal=kudu/127.13.100.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.100.20:41937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.100.62:36405,127.13.100.61:34345,127.13.100.60:44933
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707036041078-13712-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707036041078-13712-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707036041078-13712-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 18:41:59.727257 4048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:59.717727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.61:34345 (local address 127.0.0.1:50102)
0929 18:41:59.717845 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:59.717859 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:41:59.718025 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:41:59.718572 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:41:59.718574 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:41:59.718704 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:41:59.718849 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:41:59.718859 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:41:59.719866 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:41:59.719869 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:41:59.720537 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:41:59.720547 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:41:59.721298 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:41:59.721302 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:41:59.721417 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:59.721853 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:41:59.721864 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:41:59.723190 (+ 1326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:41:59.725429 (+ 2239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:41:59.725436 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:41:59.725441 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:41:59.725725 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:41:59.726118 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:41:59.726124 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:41:59.726129 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:41:59.726240 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:41:59.726855 (+ 615us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:41:59.726858 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:41:59.726948 (+ 90us) client_negotiation.cc:769] Sending connection context
0929 18:41:59.727041 (+ 93us) client_negotiation.cc:240] Negotiation successful
0929 18:41:59.727138 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260929 18:41:59.727550 4050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:59.717887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50102 (local address 127.13.100.61:34345)
0929 18:41:59.718256 (+ 369us) server_negotiation.cc:206] Beginning negotiation
0929 18:41:59.718264 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 18:41:59.718283 (+ 19us) server_negotiation.cc:407] Connection header received
0929 18:41:59.718316 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:41:59.718320 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:41:59.718455 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:41:59.718560 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:41:59.718982 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:59.719742 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:59.720701 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:59.721098 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:59.721182 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:59.723362 (+ 2180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:41:59.723370 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:41:59.723381 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:41:59.723403 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:41:59.725321 (+ 1918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:41:59.725850 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:41:59.725854 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:41:59.725855 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:41:59.726022 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:41:59.726368 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:41:59.726374 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:41:59.726377 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:41:59.726731 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:41:59.726871 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0929 18:41:59.727158 (+ 287us) server_negotiation.cc:299] Negotiation successful
0929 18:41:59.727327 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":181,"threads_started":1}
I20260929 18:41:59.729638 4045 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.100.60" port: 44933 } has no permanent_uuid. Determining permanent_uuid...
I20260929 18:41:59.730005 3970 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.100.61" port: 34345 } attempt: 3
I20260929 18:41:59.730486 4048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:59.730166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.60:44933 (local address 127.0.0.1:41404)
0929 18:41:59.730297 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:59.730390 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.60:44933: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260929 18:41:59.730760 3994 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.100.60:44933: connect: Connection refused (error 111)
I20260929 18:41:59.731675 4048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:59.731419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.60:44933 (local address 127.0.0.1:41418)
0929 18:41:59.731543 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:59.731604 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.60:44933: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260929 18:41:59.731920 4045 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.100.60:44933: Network error: Client connection negotiation failed: client connection to 127.13.100.60:44933: connect: Connection refused (error 111)
Sep 29 18:41:59 dist-test-slave-j1xs krb5kdc[3889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707319, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.62@KRBTEST.COM for kudu/127.13.100.61@KRBTEST.COM
I20260929 18:41:59.739085 3973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:59.730414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.61:34345 (local address 127.0.0.1:50118)
0929 18:41:59.730580 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:59.730592 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:41:59.730682 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:41:59.730908 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:41:59.730911 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:41:59.731108 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:41:59.731214 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:41:59.731222 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:41:59.732090 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:41:59.732094 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:41:59.733023 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:41:59.733032 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:41:59.733511 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:41:59.733515 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:41:59.733624 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:59.734114 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:41:59.734125 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:41:59.735549 (+ 1424us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:41:59.737597 (+ 2048us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:41:59.737601 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:41:59.737603 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:41:59.737800 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:41:59.738115 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:41:59.738119 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:41:59.738121 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:41:59.738205 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:41:59.738578 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:41:59.738581 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:41:59.738680 (+ 99us) client_negotiation.cc:769] Sending connection context
0929 18:41:59.738783 (+ 103us) client_negotiation.cc:240] Negotiation successful
0929 18:41:59.738917 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101}
I20260929 18:41:59.739113 4050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:59.730452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50118 (local address 127.13.100.61:34345)
0929 18:41:59.730544 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0929 18:41:59.730548 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:41:59.730654 (+ 106us) server_negotiation.cc:407] Connection header received
0929 18:41:59.730754 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:41:59.730759 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:41:59.730802 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:41:59.730883 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:41:59.731347 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:59.731975 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:59.733141 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:59.733402 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:59.733510 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:59.735693 (+ 2183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:41:59.735696 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:41:59.735699 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:41:59.735721 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:41:59.737484 (+ 1763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:41:59.737920 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:41:59.737926 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:41:59.737928 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:41:59.738012 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:41:59.738301 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:41:59.738306 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:41:59.738307 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:41:59.738476 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:41:59.738577 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 18:41:59.738841 (+ 264us) server_negotiation.cc:299] Negotiation successful
0929 18:41:59.738946 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260929 18:41:59.740024 3970 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.100.60" port: 44933 } has no permanent_uuid. Determining permanent_uuid...
I20260929 18:41:59.740703 3973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:59.740490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.60:44933 (local address 127.0.0.1:41422)
0929 18:41:59.740596 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:59.740651 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.60:44933: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260929 18:41:59.741360 3973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:59.741130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.60:44933 (local address 127.0.0.1:41434)
0929 18:41:59.741264 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:59.741308 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.60:44933: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260929 18:41:59.741473 3970 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.100.60:44933: Network error: Client connection negotiation failed: client connection to 127.13.100.60:44933: connect: Connection refused (error 111)
I20260929 18:41:59.772643 3970 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.100.60" port: 44933 } attempt: 1
I20260929 18:41:59.773588 3973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:59.773256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.60:44933 (local address 127.0.0.1:41446)
0929 18:41:59.773427 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:59.773499 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.60:44933: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260929 18:41:59.774451 3973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:59.774200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.60:44933 (local address 127.0.0.1:41460)
0929 18:41:59.774321 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:59.774375 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.60:44933: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260929 18:41:59.774626 3970 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.100.60:44933: Network error: Client connection negotiation failed: client connection to 127.13.100.60:44933: connect: Connection refused (error 111)
I20260929 18:41:59.781172 4045 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.100.60" port: 44933 } attempt: 1
I20260929 18:41:59.781853 4048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:59.781592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.60:44933 (local address 127.0.0.1:41464)
0929 18:41:59.781724 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:59.781782 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.60:44933: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260929 18:41:59.782605 4048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:59.782387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.60:44933 (local address 127.0.0.1:41480)
0929 18:41:59.782495 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:59.782545 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.60:44933: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260929 18:41:59.782781 4045 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.100.60:44933: Network error: Client connection negotiation failed: client connection to 127.13.100.60:44933: connect: Connection refused (error 111)
W20260929 18:41:59.838239 4051 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 18:41:59.838630 4051 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:41:59.838704 4051 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:41:59.838810 4051 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:41:59.842988 4051 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:41:59.843089 4051 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:41:59.843132 4051 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:41:59.843194 4051 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:41:59.843235 4051 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:41:59.848505 4051 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.100.20:41937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707036041078-13712-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707036041078-13712-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.13.100.62:36405,127.13.100.61:34345,127.13.100.60:44933
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707036041078-13712-0/krb5kdc/kudu_127.13.100.60.keytab
--principal=kudu/127.13.100.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.100.60:44933
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707036041078-13712-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.13.100.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4051
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707036041078-13712-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:41:59.849615 4051 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:41:59.849858 3970 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.100.60" port: 44933 } attempt: 2
I20260929 18:41:59.850737 3973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:59.850393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.60:44933 (local address 127.0.0.1:41496)
0929 18:41:59.850565 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:59.850643 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.60:44933: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260929 18:41:59.850814 4051 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 18:41:59.851112 4045 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.100.60" port: 44933 } attempt: 2
I20260929 18:41:59.851693 3973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:59.851431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.60:44933 (local address 127.0.0.1:41512)
0929 18:41:59.851579 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:59.851627 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.60:44933: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
W20260929 18:41:59.851832 3970 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.100.60:44933: Network error: Client connection negotiation failed: client connection to 127.13.100.60:44933: connect: Connection refused (error 111)
I20260929 18:41:59.851846 4048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:59.851527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.60:44933 (local address 127.0.0.1:41516)
0929 18:41:59.851694 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:59.851764 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.60:44933: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260929 18:41:59.852603 4048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:59.852378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.60:44933 (local address 127.0.0.1:41528)
0929 18:41:59.852490 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:59.852539 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.100.60:44933: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260929 18:41:59.852731 4045 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.100.60:44933: Network error: Client connection negotiation failed: client connection to 127.13.100.60:44933: connect: Connection refused (error 111)
W20260929 18:41:59.859675 4060 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:41:59.859675 4057 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:41:59.859743 4058 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:41:59.860086 4051 server_base.cc:1034] running on GCE node
I20260929 18:41:59.860664 4051 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:41:59.861732 4051 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:41:59.862950 4051 hybrid_clock.cc:648] HybridClock initialized: now 1790707319862915 us; error 49 us; skew 500 ppm
Sep 29 18:41:59 dist-test-slave-j1xs krb5kdc[3889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707319, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:41:59.865878 4051 init.cc:377] Logged in from keytab as kudu/127.13.100.60@KRBTEST.COM (short username kudu)
I20260929 18:41:59.866848 4051 webserver.cc:460] Webserver started at http://127.13.100.60:36459/ using document root <none> and password file <none>
I20260929 18:41:59.867374 4051 fs_manager.cc:359] Metadata directory not provided
I20260929 18:41:59.867432 4051 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:41:59.867599 4051 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:41:59.869558 4051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707036041078-13712-0/minicluster-data/master-2/data/instance:
uuid: "7f328e468156458e89726c4c6b2d8251"
format_stamp: "Formatted at 2026-09-29 18:41:59 on dist-test-slave-j1xs"
server_key: "926ecd4fb630eb2e875a3e9932b4ff4f"
server_key_iv: "052b6a80263f8d4e317428af88d2c802"
server_key_version: "encryptionkey@0"
I20260929 18:41:59.870138 4051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707036041078-13712-0/minicluster-data/master-2/wal/instance:
uuid: "7f328e468156458e89726c4c6b2d8251"
format_stamp: "Formatted at 2026-09-29 18:41:59 on dist-test-slave-j1xs"
server_key: "926ecd4fb630eb2e875a3e9932b4ff4f"
server_key_iv: "052b6a80263f8d4e317428af88d2c802"
server_key_version: "encryptionkey@0"
I20260929 18:41:59.874122 4051 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 18:41:59.876545 4067 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:41:59.878106 4051 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:41:59.878283 4051 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707036041078-13712-0/minicluster-data/master-2/data,/tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707036041078-13712-0/minicluster-data/master-2/wal
uuid: "7f328e468156458e89726c4c6b2d8251"
format_stamp: "Formatted at 2026-09-29 18:41:59 on dist-test-slave-j1xs"
server_key: "926ecd4fb630eb2e875a3e9932b4ff4f"
server_key_iv: "052b6a80263f8d4e317428af88d2c802"
server_key_version: "encryptionkey@0"
I20260929 18:41:59.878428 4051 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707036041078-13712-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707036041078-13712-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707036041078-13712-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:41:59.893349 4051 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:41:59.896726 4051 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:41:59.896901 4051 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:41:59.904347 4051 rpc_server.cc:307] RPC server started. Bound to: 127.13.100.60:44933
I20260929 18:41:59.904652 4119 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.100.60:44933 every 8 connection(s)
I20260929 18:41:59.905390 4051 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707036041078-13712-0/minicluster-data/master-2/data/info.pb
I20260929 18:41:59.908221 4120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:41:59.911821 4120 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.100.62" port: 36405 } has no permanent_uuid. Determining permanent_uuid...
I20260929 18:41:59.912127 13712 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu as pid 4051
I20260929 18:41:59.912302 13712 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpp0tHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707036041078-13712-0/minicluster-data/master-2/wal/instance
I20260929 18:41:59.912642 13712 external_mini_cluster.cc:1428] Setting key b844e7659c1ac104ad7014b3189ed565
Sep 29 18:41:59 dist-test-slave-j1xs krb5kdc[3889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707319, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.60@KRBTEST.COM for kudu/127.13.100.62@KRBTEST.COM
Sep 29 18:41:59 dist-test-slave-j1xs krb5kdc[3889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707319, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.100.60@KRBTEST.COM
I20260929 18:41:59.926257 3974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:59.913852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52978 (local address 127.13.100.62:36405)
0929 18:41:59.914007 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0929 18:41:59.914011 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:41:59.914609 (+ 598us) server_negotiation.cc:407] Connection header received
0929 18:41:59.915401 (+ 792us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:41:59.915404 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:41:59.915452 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:41:59.915592 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:41:59.916782 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:59.917275 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:59.918388 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:59.918619 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:59.918734 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:59.921642 (+ 2908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:41:59.921645 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:41:59.921647 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:41:59.921671 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:41:59.923500 (+ 1829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:41:59.924170 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:41:59.924174 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:41:59.924176 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:41:59.924260 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:41:59.924662 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:41:59.924664 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:41:59.924665 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:41:59.924957 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:41:59.925040 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 18:41:59.925732 (+ 692us) server_negotiation.cc:299] Negotiation successful
0929 18:41:59.925918 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 18:41:59.927103 4123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:59.914006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.62:36405 (local address 127.0.0.1:52978)
0929 18:41:59.914449 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:59.914504 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 18:41:59.915167 (+ 663us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:41:59.916004 (+ 837us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:41:59.916008 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:41:59.916345 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:41:59.916639 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:41:59.916652 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:41:59.917386 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:41:59.917389 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:41:59.918264 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:41:59.918274 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:41:59.918771 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:41:59.918775 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:41:59.918945 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:59.919547 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:41:59.919571 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:41:59.921457 (+ 1886us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:41:59.923624 (+ 2167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:41:59.923633 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:41:59.923643 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:41:59.924069 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:41:59.924371 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:41:59.924374 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:41:59.924375 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:41:59.924557 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:41:59.925079 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:41:59.925085 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:41:59.925476 (+ 391us) client_negotiation.cc:769] Sending connection context
0929 18:41:59.925661 (+ 185us) client_negotiation.cc:240] Negotiation successful
0929 18:41:59.926090 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"mutex_wait_us":58,"spinlock_wait_cycles":2176,"thread_start_us":135,"threads_started":1}
I20260929 18:41:59.927103 4124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:59.913997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41532 (local address 127.13.100.60:44933)
0929 18:41:59.914405 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0929 18:41:59.914420 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 18:41:59.914463 (+ 43us) server_negotiation.cc:407] Connection header received
0929 18:41:59.915151 (+ 688us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:41:59.915198 (+ 47us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:41:59.915433 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:41:59.915622 (+ 189us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:41:59.916395 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:59.917496 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:59.918574 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:59.918945 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:59.919073 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:59.922134 (+ 3061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:41:59.922140 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:41:59.922150 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:41:59.922172 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:41:59.924013 (+ 1841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:41:59.924539 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:41:59.924546 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:41:59.924548 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:41:59.924632 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:41:59.924914 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:41:59.924916 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:41:59.924917 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:41:59.925476 (+ 559us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:41:59.925686 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0929 18:41:59.925969 (+ 283us) server_negotiation.cc:299] Negotiation successful
0929 18:41:59.926140 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":89,"server-negotiator.queue_time_us":284,"spinlock_wait_cycles":34560,"thread_start_us":145,"threads_started":1}
I20260929 18:41:59.928416 4120 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.100.61" port: 34345 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 18:41:59 dist-test-slave-j1xs krb5kdc[3889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707319, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.60@KRBTEST.COM for kudu/127.13.100.61@KRBTEST.COM
Sep 29 18:41:59 dist-test-slave-j1xs krb5kdc[3889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707319, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.100.62@KRBTEST.COM
I20260929 18:41:59.937987 4123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:59.928954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.61:34345 (local address 127.0.0.1:50124)
0929 18:41:59.929065 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:59.929081 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 18:41:59.929153 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:41:59.929757 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:41:59.929760 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:41:59.930026 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:41:59.930175 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:41:59.930183 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:41:59.930980 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:41:59.930984 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:41:59.931524 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:41:59.931533 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:41:59.932060 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:41:59.932064 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:41:59.932174 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:59.932695 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:41:59.932714 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:41:59.934034 (+ 1320us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:41:59.936040 (+ 2006us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:41:59.936043 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:41:59.936044 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:41:59.936239 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:41:59.936846 (+ 607us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:41:59.936849 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:41:59.936850 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:41:59.936937 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:41:59.937408 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:41:59.937410 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:41:59.937491 (+ 81us) client_negotiation.cc:769] Sending connection context
0929 18:41:59.937618 (+ 127us) client_negotiation.cc:240] Negotiation successful
0929 18:41:59.937754 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260929 18:41:59.938023 4050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:59.928977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50124 (local address 127.13.100.61:34345)
0929 18:41:59.929105 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0929 18:41:59.929109 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:41:59.929126 (+ 17us) server_negotiation.cc:407] Connection header received
0929 18:41:59.929458 (+ 332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:41:59.929464 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:41:59.929525 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:41:59.929609 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:41:59.930319 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:59.930832 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:59.931653 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:59.931938 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:59.932064 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:59.934172 (+ 2108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:41:59.934175 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:41:59.934177 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:41:59.934199 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:41:59.935922 (+ 1723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:41:59.936340 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:41:59.936345 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:41:59.936349 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:41:59.936444 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:41:59.937046 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:41:59.937050 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:41:59.937052 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:41:59.937295 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:41:59.937408 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 18:41:59.937689 (+ 281us) server_negotiation.cc:299] Negotiation successful
0929 18:41:59.937837 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 18:41:59.938202 3974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:59.929661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52992 (local address 127.13.100.62:36405)
0929 18:41:59.929770 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0929 18:41:59.929780 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 18:41:59.929791 (+ 11us) server_negotiation.cc:407] Connection header received
0929 18:41:59.929954 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:41:59.929957 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:41:59.930000 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:41:59.930083 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:41:59.930754 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:59.931249 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:59.932006 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:59.932227 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:59.932335 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:59.934933 (+ 2598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:41:59.934940 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:41:59.934946 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:41:59.934976 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:41:59.936445 (+ 1469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:41:59.936843 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:41:59.936846 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:41:59.936847 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:41:59.936923 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:41:59.937261 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:41:59.937264 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:41:59.937266 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:41:59.937481 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:41:59.937686 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0929 18:41:59.937870 (+ 184us) server_negotiation.cc:299] Negotiation successful
0929 18:41:59.938049 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260929 18:41:59.939283 4120 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.100.60" port: 44933 } has no permanent_uuid. Determining permanent_uuid...
I20260929 18:41:59.943902 4045 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.100.60" port: 44933 } attempt: 3
Sep 29 18:41:59 dist-test-slave-j1xs krb5kdc[3889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707319, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.60@KRBTEST.COM for kudu/127.13.100.60@KRBTEST.COM
Sep 29 18:41:59 dist-test-slave-j1xs krb5kdc[3889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707319, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.100.61@KRBTEST.COM
I20260929 18:41:59.949585 4123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:59.939767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.60:44933 (local address 127.0.0.1:41538)
0929 18:41:59.939881 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:59.939894 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:41:59.939961 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:41:59.940180 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:41:59.940184 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:41:59.940381 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:41:59.940503 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:41:59.940512 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:41:59.941565 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:41:59.941568 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:41:59.942369 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:41:59.942381 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:41:59.942892 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:41:59.942896 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:41:59.942955 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:59.943431 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:41:59.943443 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:41:59.945604 (+ 2161us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:41:59.947847 (+ 2243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:41:59.947850 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:41:59.947853 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:41:59.948086 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:41:59.948400 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:41:59.948405 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:41:59.948407 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:41:59.948496 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:41:59.949011 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:41:59.949014 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:41:59.949105 (+ 91us) client_negotiation.cc:769] Sending connection context
0929 18:41:59.949230 (+ 125us) client_negotiation.cc:240] Negotiation successful
0929 18:41:59.949385 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260929 18:41:59.949615 4124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:59.939836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41538 (local address 127.13.100.60:44933)
0929 18:41:59.939925 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0929 18:41:59.939929 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:41:59.939949 (+ 20us) server_negotiation.cc:407] Connection header received
0929 18:41:59.940048 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:41:59.940049 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:41:59.940080 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:41:59.940138 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:41:59.940658 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:59.941449 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:59.942485 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:59.942776 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:59.942873 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:59.945757 (+ 2884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:41:59.945761 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:41:59.945766 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:41:59.945795 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:41:59.947718 (+ 1923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:41:59.948196 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:41:59.948199 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:41:59.948200 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:41:59.948278 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:41:59.948622 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:41:59.948625 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:41:59.948627 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:41:59.948846 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:41:59.948942 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 18:41:59.949299 (+ 357us) server_negotiation.cc:299] Negotiation successful
0929 18:41:59.949456 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":34,"server-negotiator.queue_time_us":36}
I20260929 18:41:59.950183 4050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:59.939506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50128 (local address 127.13.100.61:34345)
0929 18:41:59.939604 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0929 18:41:59.939608 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:41:59.939664 (+ 56us) server_negotiation.cc:407] Connection header received
0929 18:41:59.939872 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:41:59.939876 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:41:59.939931 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:41:59.939986 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:41:59.940885 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:59.941650 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:59.942695 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:59.943029 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:59.943106 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:59.946012 (+ 2906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:41:59.946016 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:41:59.946018 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:41:59.946041 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:41:59.947726 (+ 1685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:41:59.948556 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:41:59.948562 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:41:59.948567 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:41:59.948656 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:41:59.948962 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:41:59.948966 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:41:59.948969 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:41:59.949185 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:41:59.949471 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0929 18:41:59.949778 (+ 307us) server_negotiation.cc:299] Negotiation successful
0929 18:41:59.949923 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
Sep 29 18:41:59 dist-test-slave-j1xs krb5kdc[3889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707319, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.61@KRBTEST.COM for kudu/127.13.100.60@KRBTEST.COM
I20260929 18:41:59.950984 3970 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.100.60" port: 44933 } attempt: 3
I20260929 18:41:59.953641 13712 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:41:59.953778 4120 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f328e468156458e89726c4c6b2d8251: Bootstrap starting.
I20260929 18:41:59.955729 4048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:59.944288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.60:44933 (local address 127.0.0.1:41548)
0929 18:41:59.944405 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:59.944430 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 18:41:59.944500 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:41:59.945037 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:41:59.945041 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:41:59.945214 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:41:59.945335 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:41:59.945344 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:41:59.947217 (+ 1873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:41:59.947223 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:41:59.948007 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:41:59.948017 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:41:59.948637 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:41:59.948640 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:41:59.948727 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:59.949180 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:41:59.949201 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:41:59.951094 (+ 1893us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:41:59.953972 (+ 2878us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:41:59.953975 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:41:59.953978 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:41:59.954232 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:41:59.954591 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:41:59.954595 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:41:59.954597 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:41:59.954690 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:41:59.955211 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:41:59.955214 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:41:59.955313 (+ 99us) client_negotiation.cc:769] Sending connection context
0929 18:41:59.955418 (+ 105us) client_negotiation.cc:240] Negotiation successful
0929 18:41:59.955554 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260929 18:41:59.954936 3944 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "667b13f021124ab7a5cf950563a3cd31"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52992
I20260929 18:41:59.956435 4125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:59.944456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41548 (local address 127.13.100.60:44933)
0929 18:41:59.944824 (+ 368us) server_negotiation.cc:206] Beginning negotiation
0929 18:41:59.944831 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 18:41:59.944841 (+ 10us) server_negotiation.cc:407] Connection header received
0929 18:41:59.944867 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:41:59.944869 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:41:59.944899 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:41:59.944969 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:41:59.945457 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:59.946604 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:59.948122 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:59.948504 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:59.948630 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:59.952007 (+ 3377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:41:59.952011 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:41:59.952014 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:41:59.952038 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:41:59.953801 (+ 1763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:41:59.954380 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:41:59.954384 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:41:59.954386 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:41:59.954476 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:41:59.954813 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:41:59.954818 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:41:59.954822 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:41:59.955077 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:41:59.956025 (+ 948us) server_negotiation.cc:1035] Waiting for connection context
0929 18:41:59.956109 (+ 84us) server_negotiation.cc:299] Negotiation successful
0929 18:41:59.956235 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":152,"threads_started":1}
Sep 29 18:41:59 dist-test-slave-j1xs krb5kdc[3889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707319, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.100.62@KRBTEST.COM for kudu/127.13.100.60@KRBTEST.COM
I20260929 18:41:59.958382 4120 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7f328e468156458e89726c4c6b2d8251: Neither blocks nor log segments found. Creating new log.
I20260929 18:41:59.959136 4120 log.cc:824] T 00000000000000000000000000000000 P 7f328e468156458e89726c4c6b2d8251: Log is configured to *not* fsync() on all Append() calls
I20260929 18:41:59.959745 4045 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af2dec8be43940c7823a2d8268ddaf47: Bootstrap starting.
I20260929 18:41:59.960845 3973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:59.951438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.100.60:44933 (local address 127.0.0.1:41562)
0929 18:41:59.951754 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0929 18:41:59.951770 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 18:41:59.951855 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:41:59.952174 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:41:59.952177 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:41:59.952653 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:41:59.952775 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:41:59.952783 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:41:59.953994 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:41:59.953997 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:41:59.954746 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:41:59.954756 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:41:59.955194 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:41:59.955198 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:41:59.955281 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:59.955745 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:41:59.955760 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:41:59.957173 (+ 1413us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:41:59.959128 (+ 1955us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:41:59.959131 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:41:59.959133 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:41:59.959383 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:41:59.959695 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:41:59.959701 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:41:59.959706 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:41:59.959815 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:41:59.960282 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:41:59.960286 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:41:59.960400 (+ 114us) client_negotiation.cc:769] Sending connection context
0929 18:41:59.960517 (+ 117us) client_negotiation.cc:240] Negotiation successful
0929 18:41:59.960676 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191}
I20260929 18:41:59.960845 4124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:59.951765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41562 (local address 127.13.100.60:44933)
0929 18:41:59.951874 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0929 18:41:59.951878 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:41:59.951898 (+ 20us) server_negotiation.cc:407] Connection header received
0929 18:41:59.951934 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:41:59.951937 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:41:59.951980 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:41:59.952049 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:41:59.953156 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:59.953818 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:59.954844 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:59.955074 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:59.955182 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:59.957315 (+ 2133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:41:59.957318 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:41:59.957321 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:41:59.957346 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:41:59.959004 (+ 1658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:41:59.959497 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:41:59.959500 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:41:59.959502 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:41:59.959581 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:41:59.959972 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:41:59.959979 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:41:59.959983 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:41:59.960162 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:41:59.960279 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0929 18:41:59.960581 (+ 302us) server_negotiation.cc:299] Negotiation successful
0929 18:41:59.960703 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 18:41:59.961529 4120 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f328e468156458e89726c4c6b2d8251: No bootstrap required, opened a new log
I20260929 18:41:59.964105 3970 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 667b13f021124ab7a5cf950563a3cd31: Bootstrap starting.
I20260929 18:41:59.965531 4120 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7f328e468156458e89726c4c6b2d8251 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "667b13f021124ab7a5cf950563a3cd31" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 36405 } } peers { permanent_uuid: "af2dec8be43940c7823a2d8268ddaf47" member_type: VOTER last_known_addr { host: "127.13.100.61" port: 34345 } } peers { permanent_uuid: "7f328e468156458e89726c4c6b2d8251" member_type: VOTER last_known_addr { host: "127.13.100.60" port: 44933 } }
I20260929 18:41:59.965771 4120 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7f328e468156458e89726c4c6b2d8251 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f328e468156458e89726c4c6b2d8251, State: Initialized, Role: FOLLOWER
I20260929 18:41:59.966193 4120 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7f328e468156458e89726c4c6b2d8251 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "667b13f021124ab7a5cf950563a3cd31" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 36405 } } peers { permanent_uuid: "af2dec8be43940c7823a2d8268ddaf47" member_type: VOTER last_known_addr { host: "127.13.100.61" port: 34345 } } peers { permanent_uuid: "7f328e468156458e89726c4c6b2d8251" member_type: VOTER last_known_addr { host: "127.13.100.60" port: 44933 } }
I20260929 18:41:59.966455 4045 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P af2dec8be43940c7823a2d8268ddaf47: Neither blocks nor log segments found. Creating new log.
I20260929 18:41:59.966542 3970 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 667b13f021124ab7a5cf950563a3cd31: Neither blocks nor log segments found. Creating new log.
I20260929 18:41:59.966954 4127 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f328e468156458e89726c4c6b2d8251 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "667b13f021124ab7a5cf950563a3cd31" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 36405 } } peers { permanent_uuid: "af2dec8be43940c7823a2d8268ddaf47" member_type: VOTER last_known_addr { host: "127.13.100.61" port: 34345 } } peers { permanent_uuid: "7f328e468156458e89726c4c6b2d8251" member_type: VOTER last_known_addr { host: "127.13.100.60" port: 44933 } } }
I20260929 18:41:59.967305 4127 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f328e468156458e89726c4c6b2d8251 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:41:59.967343 3970 log.cc:824] T 00000000000000000000000000000000 P 667b13f021124ab7a5cf950563a3cd31: Log is configured to *not* fsync() on all Append() calls
I20260929 18:41:59.967344 4045 log.cc:824] T 00000000000000000000000000000000 P af2dec8be43940c7823a2d8268ddaf47: Log is configured to *not* fsync() on all Append() calls
I20260929 18:41:59.967634 4120 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7f328e468156458e89726c4c6b2d8251 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:41:59.969269 3970 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 667b13f021124ab7a5cf950563a3cd31: No bootstrap required, opened a new log
I20260929 18:41:59.969537 4045 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af2dec8be43940c7823a2d8268ddaf47: No bootstrap required, opened a new log
I20260929 18:41:59.972462 3970 raft_consensus.cc:348] T 00000000000000000000000000000000 P 667b13f021124ab7a5cf950563a3cd31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "667b13f021124ab7a5cf950563a3cd31" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 36405 } } peers { permanent_uuid: "af2dec8be43940c7823a2d8268ddaf47" member_type: VOTER last_known_addr { host: "127.13.100.61" port: 34345 } } peers { permanent_uuid: "7f328e468156458e89726c4c6b2d8251" member_type: VOTER last_known_addr { host: "127.13.100.60" port: 44933 } }
I20260929 18:41:59.972683 3970 raft_consensus.cc:729] T 00000000000000000000000000000000 P 667b13f021124ab7a5cf950563a3cd31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 667b13f021124ab7a5cf950563a3cd31, State: Initialized, Role: FOLLOWER
I20260929 18:41:59.973016 3970 consensus_queue.cc:260] T 00000000000000000000000000000000 P 667b13f021124ab7a5cf950563a3cd31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "667b13f021124ab7a5cf950563a3cd31" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 36405 } } peers { permanent_uuid: "af2dec8be43940c7823a2d8268ddaf47" member_type: VOTER last_known_addr { host: "127.13.100.61" port: 34345 } } peers { permanent_uuid: "7f328e468156458e89726c4c6b2d8251" member_type: VOTER last_known_addr { host: "127.13.100.60" port: 44933 } }
W20260929 18:41:59.973412 4140 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7f328e468156458e89726c4c6b2d8251: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:41:59.973526 4140 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:41:59.974154 4045 raft_consensus.cc:348] T 00000000000000000000000000000000 P af2dec8be43940c7823a2d8268ddaf47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "667b13f021124ab7a5cf950563a3cd31" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 36405 } } peers { permanent_uuid: "af2dec8be43940c7823a2d8268ddaf47" member_type: VOTER last_known_addr { host: "127.13.100.61" port: 34345 } } peers { permanent_uuid: "7f328e468156458e89726c4c6b2d8251" member_type: VOTER last_known_addr { host: "127.13.100.60" port: 44933 } }
I20260929 18:41:59.974386 4045 raft_consensus.cc:729] T 00000000000000000000000000000000 P af2dec8be43940c7823a2d8268ddaf47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af2dec8be43940c7823a2d8268ddaf47, State: Initialized, Role: FOLLOWER
I20260929 18:41:59.974707 3944 raft_consensus.cc:482] T 00000000000000000000000000000000 P 667b13f021124ab7a5cf950563a3cd31 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 18:41:59.974694 4045 consensus_queue.cc:260] T 00000000000000000000000000000000 P af2dec8be43940c7823a2d8268ddaf47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "667b13f021124ab7a5cf950563a3cd31" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 36405 } } peers { permanent_uuid: "af2dec8be43940c7823a2d8268ddaf47" member_type: VOTER last_known_addr { host: "127.13.100.61" port: 34345 } } peers { permanent_uuid: "7f328e468156458e89726c4c6b2d8251" member_type: VOTER last_known_addr { host: "127.13.100.60" port: 44933 } }
I20260929 18:41:59.974829 3944 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 667b13f021124ab7a5cf950563a3cd31 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:41:59.971598 4051 master_runner.cc:186] Error getting master registration for 127.13.100.62:36405: OK, instance_id { permanent_uuid: "667b13f021124ab7a5cf950563a3cd31" instance_seqno: 1790707319485856 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 18:41:59.975235 4051 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 18:41:59.975829 4045 sys_catalog.cc:564] T 00000000000000000000000000000000 P af2dec8be43940c7823a2d8268ddaf47 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:41:59.976115 3944 raft_consensus.cc:504] T 00000000000000000000000000000000 P 667b13f021124ab7a5cf950563a3cd31 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "667b13f021124ab7a5cf950563a3cd31" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 36405 } } peers { permanent_uuid: "af2dec8be43940c7823a2d8268ddaf47" member_type: VOTER last_known_addr { host: "127.13.100.61" port: 34345 } } peers { permanent_uuid: "7f328e468156458e89726c4c6b2d8251" member_type: VOTER last_known_addr { host: "127.13.100.60" port: 44933 } }
I20260929 18:41:59.977036 3944 leader_election.cc:288] T 00000000000000000000000000000000 P 667b13f021124ab7a5cf950563a3cd31 [CANDIDATE]: Term 1 election: Requested vote from peers af2dec8be43940c7823a2d8268ddaf47 (127.13.100.61:34345), 7f328e468156458e89726c4c6b2d8251 (127.13.100.60:44933)
I20260929 18:41:59.977591 4019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "667b13f021124ab7a5cf950563a3cd31" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "af2dec8be43940c7823a2d8268ddaf47"
I20260929 18:41:59.977696 4094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "667b13f021124ab7a5cf950563a3cd31" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "7f328e468156458e89726c4c6b2d8251"
I20260929 18:41:59.977918 4094 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7f328e468156458e89726c4c6b2d8251 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:41:59.977845 4019 raft_consensus.cc:3037] T 00000000000000000000000000000000 P af2dec8be43940c7823a2d8268ddaf47 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:41:59.979256 4094 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7f328e468156458e89726c4c6b2d8251 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 667b13f021124ab7a5cf950563a3cd31 in term 1.
I20260929 18:41:59.979682 3919 leader_election.cc:302] T 00000000000000000000000000000000 P 667b13f021124ab7a5cf950563a3cd31 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 667b13f021124ab7a5cf950563a3cd31, 7f328e468156458e89726c4c6b2d8251; no voters:
I20260929 18:41:59.980017 4019 raft_consensus.cc:2454] T 00000000000000000000000000000000 P af2dec8be43940c7823a2d8268ddaf47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 667b13f021124ab7a5cf950563a3cd31 in term 1.
I20260929 18:41:59.974633 3970 sys_catalog.cc:564] T 00000000000000000000000000000000 P 667b13f021124ab7a5cf950563a3cd31 [sys.catalog]: configured and running, proceeding with master startup.
W20260929 18:41:59.982507 3923 tablet.cc:2369] T 00000000000000000000000000000000 P 667b13f021124ab7a5cf950563a3cd31: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:41:59.979348 3977 master_runner.cc:186] Error getting master registration for 127.13.100.62:36405: OK, instance_id { permanent_uuid: "667b13f021124ab7a5cf950563a3cd31" instance_seqno: 1790707319485856 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 18:41:59.983278 3977 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 18:41:59.983981 4143 sys_catalog.cc:455] T 00000000000000000000000000000000 P af2dec8be43940c7823a2d8268ddaf47 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "667b13f021124ab7a5cf950563a3cd31" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 36405 } } peers { permanent_uuid: "af2dec8be43940c7823a2d8268ddaf47" member_type: VOTER last_known_addr { host: "127.13.100.61" port: 34345 } } peers { permanent_uuid: "7f328e468156458e89726c4c6b2d8251" member_type: VOTER last_known_addr { host: "127.13.100.60" port: 44933 } } }
I20260929 18:41:59.984139 4143 sys_catalog.cc:458] T 00000000000000000000000000000000 P af2dec8be43940c7823a2d8268ddaf47 [sys.catalog]: This master's current role is: FOLLOWER
W20260929 18:41:59.986361 4154 catalog_manager.cc:1558] T 00000000000000000000000000000000 P af2dec8be43940c7823a2d8268ddaf47: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260929 18:41:59.986873 4155 sys_catalog.cc:455] T 00000000000000000000000000000000 P 667b13f021124ab7a5cf950563a3cd31 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "667b13f021124ab7a5cf950563a3cd31" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 36405 } } peers { permanent_uuid: "af2dec8be43940c7823a2d8268ddaf47" member_type: VOTER last_known_addr { host: "127.13.100.61" port: 34345 } } peers { permanent_uuid: "7f328e468156458e89726c4c6b2d8251" member_type: VOTER last_known_addr { host: "127.13.100.60" port: 44933 } } }
I20260929 18:41:59.986999 4155 sys_catalog.cc:458] T 00000000000000000000000000000000 P 667b13f021124ab7a5cf950563a3cd31 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:41:59.987087 4155 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 667b13f021124ab7a5cf950563a3cd31 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:41:59.987169 4155 raft_consensus.cc:686] T 00000000000000000000000000000000 P 667b13f021124ab7a5cf950563a3cd31 [term 1 LEADER]: Becoming Leader. State: Replica: 667b13f021124ab7a5cf950563a3cd31, State: Running, Role: LEADER
W20260929 18:41:59.987494 4171 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 667b13f021124ab7a5cf950563a3cd31: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:41:59.987588 4171 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:41:59.987506 4155 consensus_queue.cc:237] T 00000000000000000000000000000000 P 667b13f021124ab7a5cf950563a3cd31 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "667b13f021124ab7a5cf950563a3cd31" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 36405 } } peers { permanent_uuid: "af2dec8be43940c7823a2d8268ddaf47" member_type: VOTER last_known_addr { host: "127.13.100.61" port: 34345 } } peers { permanent_uuid: "7f328e468156458e89726c4c6b2d8251" member_type: VOTER last_known_addr { host: "127.13.100.60" port: 44933 } }
W20260929 18:41:59.989849 4154 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:41:59.990955 3902 master_runner.cc:263] No leader master found from master 667b13f021124ab7a5cf950563a3cd31
I20260929 18:41:59.991029 3902 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 18:41:59.991565 4155 sys_catalog.cc:455] T 00000000000000000000000000000000 P 667b13f021124ab7a5cf950563a3cd31 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 667b13f021124ab7a5cf950563a3cd31. Latest consensus state: current_term: 1 leader_uuid: "667b13f021124ab7a5cf950563a3cd31" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "667b13f021124ab7a5cf950563a3cd31" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 36405 } } peers { permanent_uuid: "af2dec8be43940c7823a2d8268ddaf47" member_type: VOTER last_known_addr { host: "127.13.100.61" port: 34345 } } peers { permanent_uuid: "7f328e468156458e89726c4c6b2d8251" member_type: VOTER last_known_addr { host: "127.13.100.60" port: 44933 } } }
I20260929 18:41:59.991681 4155 sys_catalog.cc:458] T 00000000000000000000000000000000 P 667b13f021124ab7a5cf950563a3cd31 [sys.catalog]: This master's current role is: LEADER
I20260929 18:41:59.998512 3974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:59.983408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52996 (local address 127.13.100.62:36405)
0929 18:41:59.983549 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0929 18:41:59.983555 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:41:59.985433 (+ 1878us) server_negotiation.cc:407] Connection header received
0929 18:41:59.985616 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:41:59.985619 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:41:59.985663 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:41:59.985714 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:41:59.987591 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:59.988388 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:59.991936 (+ 3548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:59.992314 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:59.992395 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:59.994495 (+ 2100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:41:59.994500 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:41:59.994503 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:41:59.994533 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:41:59.996380 (+ 1847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:41:59.996937 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:41:59.996941 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:41:59.996946 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:41:59.997040 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:41:59.997377 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:41:59.997381 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:41:59.997383 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:41:59.997612 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:41:59.997751 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0929 18:41:59.998197 (+ 446us) server_negotiation.cc:299] Negotiation successful
0929 18:41:59.998332 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 18:42:00.002713 4050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:59.984620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50132 (local address 127.13.100.61:34345)
0929 18:41:59.984718 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0929 18:41:59.984722 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:41:59.993212 (+ 8490us) server_negotiation.cc:407] Connection header received
0929 18:41:59.993391 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:41:59.993394 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:41:59.993444 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:41:59.993500 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:41:59.995054 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:59.995815 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:59.996969 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:41:59.997299 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:41:59.997368 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:41:59.999074 (+ 1706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:41:59.999078 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:41:59.999081 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:41:59.999109 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:42:00.000781 (+ 1672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:00.001281 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:00.001285 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:00.001287 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:00.001374 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:00.001685 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:00.001689 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:00.001690 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:00.001937 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:42:00.002079 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:00.002432 (+ 353us) server_negotiation.cc:299] Negotiation successful
0929 18:42:00.002549 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260929 18:42:00.003336 4177 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:42:00.004506 4177 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:42:00.008340 4124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:41:59.990290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41566 (local address 127.13.100.60:44933)
0929 18:41:59.990415 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0929 18:41:59.990419 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:41:59.999198 (+ 8779us) server_negotiation.cc:407] Connection header received
0929 18:41:59.999401 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:41:59.999405 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:41:59.999460 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:41:59.999521 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:00.000371 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:00.001136 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:00.002270 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:00.002595 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:00.002677 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:00.004412 (+ 1735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:42:00.004417 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:42:00.004419 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:42:00.004449 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:42:00.006350 (+ 1901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:00.006894 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:00.006898 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:00.006900 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:00.006994 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:00.007325 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:00.007329 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:00.007330 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:00.007551 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:42:00.007654 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:00.008039 (+ 385us) server_negotiation.cc:299] Negotiation successful
0929 18:42:00.008170 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 18:42:00.013754 4094 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7f328e468156458e89726c4c6b2d8251 [term 1 FOLLOWER]: Refusing update from remote peer 667b13f021124ab7a5cf950563a3cd31: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:42:00.013754 4019 raft_consensus.cc:1261] T 00000000000000000000000000000000 P af2dec8be43940c7823a2d8268ddaf47 [term 1 FOLLOWER]: Refusing update from remote peer 667b13f021124ab7a5cf950563a3cd31: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:42:00.014510 4155 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 667b13f021124ab7a5cf950563a3cd31 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f328e468156458e89726c4c6b2d8251" member_type: VOTER last_known_addr { host: "127.13.100.60" port: 44933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:42:00.014967 4155 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 667b13f021124ab7a5cf950563a3cd31 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af2dec8be43940c7823a2d8268ddaf47" member_type: VOTER last_known_addr { host: "127.13.100.61" port: 34345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:42:00.018994 4143 sys_catalog.cc:455] T 00000000000000000000000000000000 P af2dec8be43940c7823a2d8268ddaf47 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 667b13f021124ab7a5cf950563a3cd31. Latest consensus state: current_term: 1 leader_uuid: "667b13f021124ab7a5cf950563a3cd31" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "667b13f021124ab7a5cf950563a3cd31" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 36405 } } peers { permanent_uuid: "af2dec8be43940c7823a2d8268ddaf47" member_type: VOTER last_known_addr { host: "127.13.100.61" port: 34345 } } peers { permanent_uuid: "7f328e468156458e89726c4c6b2d8251" member_type: VOTER last_known_addr { host: "127.13.100.60" port: 44933 } } }
I20260929 18:42:00.019114 4143 sys_catalog.cc:458] T 00000000000000000000000000000000 P af2dec8be43940c7823a2d8268ddaf47 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:42:00.019416 4127 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f328e468156458e89726c4c6b2d8251 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 667b13f021124ab7a5cf950563a3cd31. Latest consensus state: current_term: 1 leader_uuid: "667b13f021124ab7a5cf950563a3cd31" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "667b13f021124ab7a5cf950563a3cd31" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 36405 } } peers { permanent_uuid: "af2dec8be43940c7823a2d8268ddaf47" member_type: VOTER last_known_addr { host: "127.13.100.61" port: 34345 } } peers { permanent_uuid: "7f328e468156458e89726c4c6b2d8251" member_type: VOTER last_known_addr { host: "127.13.100.60" port: 44933 } } }
I20260929 18:42:00.019536 4127 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f328e468156458e89726c4c6b2d8251 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:42:00.025506 4142 sys_catalog.cc:455] T 00000000000000000000000000000000 P 667b13f021124ab7a5cf950563a3cd31 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "667b13f021124ab7a5cf950563a3cd31" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "667b13f021124ab7a5cf950563a3cd31" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 36405 } } peers { permanent_uuid: "af2dec8be43940c7823a2d8268ddaf47" member_type: VOTER last_known_addr { host: "127.13.100.61" port: 34345 } } peers { permanent_uuid: "7f328e468156458e89726c4c6b2d8251" member_type: VOTER last_known_addr { host: "127.13.100.60" port: 44933 } } }
I20260929 18:42:00.025669 4142 sys_catalog.cc:458] T 00000000000000000000000000000000 P 667b13f021124ab7a5cf950563a3cd31 [sys.catalog]: This master's current role is: LEADER
I20260929 18:42:00.026861 4143 sys_catalog.cc:455] T 00000000000000000000000000000000 P af2dec8be43940c7823a2d8268ddaf47 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "667b13f021124ab7a5cf950563a3cd31" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "667b13f021124ab7a5cf950563a3cd31" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 36405 } } peers { permanent_uuid: "af2dec8be43940c7823a2d8268ddaf47" member_type: VOTER last_known_addr { host: "127.13.100.61" port: 34345 } } peers { permanent_uuid: "7f328e468156458e89726c4c6b2d8251" member_type: VOTER last_known_addr { host: "127.13.100.60" port: 44933 } } }
I20260929 18:42:00.026994 4143 sys_catalog.cc:458] T 00000000000000000000000000000000 P af2dec8be43940c7823a2d8268ddaf47 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:42:00.027463 4178 mvcc.cc:205] Tried to move back new op lower bound from 7334737182767403008 to 7334737182681346048. Current Snapshot: MvccSnapshot[applied={T|T < 7334737182767403008}]
I20260929 18:42:00.030057 4155 sys_catalog.cc:455] T 00000000000000000000000000000000 P 667b13f021124ab7a5cf950563a3cd31 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "667b13f021124ab7a5cf950563a3cd31" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "667b13f021124ab7a5cf950563a3cd31" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 36405 } } peers { permanent_uuid: "af2dec8be43940c7823a2d8268ddaf47" member_type: VOTER last_known_addr { host: "127.13.100.61" port: 34345 } } peers { permanent_uuid: "7f328e468156458e89726c4c6b2d8251" member_type: VOTER last_known_addr { host: "127.13.100.60" port: 44933 } } }
I20260929 18:42:00.030207 4155 sys_catalog.cc:458] T 00000000000000000000000000000000 P 667b13f021124ab7a5cf950563a3cd31 [sys.catalog]: This master's current role is: LEADER
I20260929 18:42:00.031271 4127 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f328e468156458e89726c4c6b2d8251 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "667b13f021124ab7a5cf950563a3cd31" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "667b13f021124ab7a5cf950563a3cd31" member_type: VOTER last_known_addr { host: "127.13.100.62" port: 36405 } } peers { permanent_uuid: "af2dec8be43940c7823a2d8268ddaf47" member_type: VOTER last_known_addr { host: "127.13.100.61" port: 34345 } } peers { permanent_uuid: "7f328e468156458e89726c4c6b2d8251" member_type: VOTER last_known_addr { host: "127.13.100.60" port: 44933 } } }
I20260929 18:42:00.031399 4127 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f328e468156458e89726c4c6b2d8251 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:42:00.031998 4179 mvcc.cc:205] Tried to move back new op lower bound from 7334737182767403008 to 7334737182681346048. Current Snapshot: MvccSnapshot[applied={T|T < 7334737182767403008}]
I20260929 18:42:00.036935 4177 catalog_manager.cc:1347] Generated new cluster ID: 8df0884d8a6f471e88ff20c22125424a
I20260929 18:42:00.037034 4177 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:42:00.069643 4177 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:42:00.071105 4177 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:42:00.095518 4177 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 667b13f021124ab7a5cf950563a3cd31: Generated new TSK 0
I20260929 18:42:00.096474 4177 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 18:42:00.152050 4124 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:42:00.148791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41576 (local address 127.13.100.60:44933)
0929 18:42:00.148936 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:00.148944 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:00.150879 (+ 1935us) server_negotiation.cc:407] Connection header received
0929 18:42:00.151097 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:00.151101 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:00.151155 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:00.151214 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:00.151913 (+ 699us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41576: BlockingRecv error: recv got EOF from 127.0.0.1:41576 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260929 18:42:00.153519 3974 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:42:00.149168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53000 (local address 127.13.100.62:36405)
0929 18:42:00.149288 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:00.149293 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:00.152578 (+ 3285us) server_negotiation.cc:407] Connection header received
0929 18:42:00.152765 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:00.152769 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:00.152821 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:00.152878 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:00.153399 (+ 521us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53000: BlockingRecv error: recv got EOF from 127.0.0.1:53000 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260929 18:42:00.154917 4050 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:42:00.149621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50140 (local address 127.13.100.61:34345)
0929 18:42:00.149738 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:00.149744 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:00.154023 (+ 4279us) server_negotiation.cc:407] Connection header received
0929 18:42:00.154201 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:00.154205 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:00.154256 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:00.154309 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:00.154803 (+ 494us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50140: BlockingRecv error: recv got EOF from 127.0.0.1:50140 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260929 18:42:00.159768 4050 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:42:00.158159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50146 (local address 127.13.100.61:34345)
0929 18:42:00.158269 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:00.158274 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:00.158763 (+ 489us) server_negotiation.cc:407] Connection header received
0929 18:42:00.159007 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:00.159011 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:00.159084 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:00.159142 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:00.159670 (+ 528us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50146: BlockingRecv error: recv got EOF from 127.0.0.1:50146 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260929 18:42:00.164965 4050 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:42:00.163331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50158 (local address 127.13.100.61:34345)
0929 18:42:00.163449 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:00.163453 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:00.164009 (+ 556us) server_negotiation.cc:407] Connection header received
0929 18:42:00.164200 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:00.164205 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:00.164260 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:00.164317 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:00.164849 (+ 532us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50158: BlockingRecv error: recv got EOF from 127.0.0.1:50158 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260929 18:42:00.173331 4050 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:42:00.170333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50172 (local address 127.13.100.61:34345)
0929 18:42:00.170460 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:00.170467 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:00.172299 (+ 1832us) server_negotiation.cc:407] Connection header received
0929 18:42:00.172550 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:00.172553 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:00.172606 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:00.172661 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:00.173218 (+ 557us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50172: BlockingRecv error: recv got EOF from 127.0.0.1:50172 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260929 18:42:00.183758 4050 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:42:00.181850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50186 (local address 127.13.100.61:34345)
0929 18:42:00.182028 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:00.182033 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:00.182701 (+ 668us) server_negotiation.cc:407] Connection header received
0929 18:42:00.182902 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:00.182906 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:00.182969 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:00.183040 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:00.183642 (+ 602us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50186: BlockingRecv error: recv got EOF from 127.0.0.1:50186 (error 108)
Metrics: {"server-negotiator.queue_time_us":105}
W20260929 18:42:00.196378 4050 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:42:00.194403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50190 (local address 127.13.100.61:34345)
0929 18:42:00.194533 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:00.194538 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:00.195242 (+ 704us) server_negotiation.cc:407] Connection header received
0929 18:42:00.195497 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:00.195501 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:00.195563 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:00.195625 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:00.196241 (+ 616us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50190: BlockingRecv error: recv got EOF from 127.0.0.1:50190 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260929 18:42:00.216675 4050 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:42:00.215229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50206 (local address 127.13.100.61:34345)
0929 18:42:00.215393 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:00.215400 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:00.215656 (+ 256us) server_negotiation.cc:407] Connection header received
0929 18:42:00.215782 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:00.215785 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:00.215842 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:00.215943 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:00.216523 (+ 580us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50206: BlockingRecv error: recv got EOF from 127.0.0.1:50206 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260929 18:42:00.251622 4050 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:42:00.250525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50220 (local address 127.13.100.61:34345)
0929 18:42:00.250679 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:00.250683 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:00.250761 (+ 78us) server_negotiation.cc:407] Connection header received
0929 18:42:00.250884 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:00.250887 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:00.250933 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:00.251004 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:00.251460 (+ 456us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50220: BlockingRecv error: recv got EOF from 127.0.0.1:50220 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260929 18:42:00.318998 4050 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:42:00.317742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50230 (local address 127.13.100.61:34345)
0929 18:42:00.317945 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:00.317949 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:00.318100 (+ 151us) server_negotiation.cc:407] Connection header received
0929 18:42:00.318282 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:00.318291 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:00.318339 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:00.318432 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:00.318892 (+ 460us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50230: BlockingRecv error: recv got EOF from 127.0.0.1:50230 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
W20260929 18:42:00.450771 4050 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:42:00.449343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50246 (local address 127.13.100.61:34345)
0929 18:42:00.449510 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:00.449514 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:00.449635 (+ 121us) server_negotiation.cc:407] Connection header received
0929 18:42:00.449758 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:00.449760 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:00.449807 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:00.450030 (+ 223us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:00.450615 (+ 585us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50246: BlockingRecv error: recv got EOF from 127.0.0.1:50246 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
W20260929 18:42:00.710258 4050 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:42:00.709026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50262 (local address 127.13.100.61:34345)
0929 18:42:00.709210 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:00.709216 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:00.709234 (+ 18us) server_negotiation.cc:407] Connection header received
0929 18:42:00.709403 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:00.709407 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:00.709457 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:00.709554 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:00.710134 (+ 580us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50262: BlockingRecv error: recv got EOF from 127.0.0.1:50262 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
I20260929 18:42:00.974786 4140 catalog_manager.cc:1259] Loaded cluster ID: 8df0884d8a6f471e88ff20c22125424a
I20260929 18:42:00.974896 4140 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7f328e468156458e89726c4c6b2d8251: loading cluster ID for follower catalog manager: success
I20260929 18:42:00.976677 4140 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7f328e468156458e89726c4c6b2d8251: acquiring CA information for follower catalog manager: success
I20260929 18:42:00.977958 4140 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7f328e468156458e89726c4c6b2d8251: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 18:42:00.991103 4154 catalog_manager.cc:1259] Loaded cluster ID: 8df0884d8a6f471e88ff20c22125424a
I20260929 18:42:00.991206 4154 catalog_manager.cc:1552] T 00000000000000000000000000000000 P af2dec8be43940c7823a2d8268ddaf47: loading cluster ID for follower catalog manager: success
I20260929 18:42:00.992805 4154 catalog_manager.cc:1574] T 00000000000000000000000000000000 P af2dec8be43940c7823a2d8268ddaf47: acquiring CA information for follower catalog manager: success
I20260929 18:42:00.993964 4154 catalog_manager.cc:1602] T 00000000000000000000000000000000 P af2dec8be43940c7823a2d8268ddaf47: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 18:42:01.228202 4250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:01.224327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50264 (local address 127.13.100.61:34345)
0929 18:42:01.224610 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:01.224615 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:01.224633 (+ 18us) server_negotiation.cc:407] Connection header received
0929 18:42:01.224729 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:01.224732 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:01.224871 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:01.224980 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:42:01.225255 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:01.225802 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:01.226703 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:01.226952 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:01.227036 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:01.227344 (+ 308us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:42:01.227777 (+ 433us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:42:01.227869 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:01.227945 (+ 76us) server_negotiation.cc:299] Negotiation successful
0929 18:42:01.227995 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":72,"threads_started":1}
I20260929 18:42:01.234990 4256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:01.231387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41578 (local address 127.13.100.60:44933)
0929 18:42:01.231614 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:01.231617 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:01.231629 (+ 12us) server_negotiation.cc:407] Connection header received
0929 18:42:01.231699 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:01.231702 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:01.231802 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:01.231898 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:42:01.232181 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:01.232695 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:01.233545 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:01.233766 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:01.233865 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:01.234181 (+ 316us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:42:01.234618 (+ 437us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:42:01.234734 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:01.234807 (+ 73us) server_negotiation.cc:299] Negotiation successful
0929 18:42:01.234863 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":88,"threads_started":1}
I20260929 18:42:01.242004 4263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:01.237895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53012 (local address 127.13.100.62:36405)
0929 18:42:01.238137 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:01.238141 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:01.238156 (+ 15us) server_negotiation.cc:407] Connection header received
0929 18:42:01.238303 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:01.238309 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:01.238437 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:01.238574 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:42:01.238785 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:01.239471 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:01.240477 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:01.240714 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:01.240819 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:01.241119 (+ 300us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:42:01.241392 (+ 273us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:42:01.241490 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:01.241753 (+ 263us) server_negotiation.cc:299] Negotiation successful
0929 18:42:01.241817 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":72,"threads_started":1}
I20260929 18:42:01.242137 4250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:01.237737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50272 (local address 127.13.100.61:34345)
0929 18:42:01.237863 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:01.237867 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:01.238478 (+ 611us) server_negotiation.cc:407] Connection header received
0929 18:42:01.238642 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:01.238644 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:01.238691 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:01.238737 (+ 46us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:42:01.239152 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:01.239718 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:01.240991 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:01.241243 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:01.241322 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:01.241608 (+ 286us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:42:01.241692 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:42:01.241762 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:01.241945 (+ 183us) server_negotiation.cc:299] Negotiation successful
0929 18:42:01.241995 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 18:42:01.242977 4256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:01.238084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41588 (local address 127.13.100.60:44933)
0929 18:42:01.238186 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:01.238190 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:01.238491 (+ 301us) server_negotiation.cc:407] Connection header received
0929 18:42:01.238641 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:01.238644 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:01.238691 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:01.238737 (+ 46us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:42:01.239116 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:01.239754 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:01.241376 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:01.241682 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:01.241774 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:01.242127 (+ 353us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:42:01.242195 (+ 68us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:42:01.242237 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:01.242770 (+ 533us) server_negotiation.cc:299] Negotiation successful
0929 18:42:01.242855 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 18:42:01.245415 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 3902
I20260929 18:42:01.251534 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 3977
I20260929 18:42:01.257251 13712 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpp0tHW/build/debug/bin/kudu with pid 4051
2026-09-29T18:42:01Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3100 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6207 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (285211 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST